Rendered at 12:57:57 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
k1m 2 days ago [-]
I agree with Roy Fielding on this:
> It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know which image format would be usable on all UAs and there was no CSS or javascript to allow for client-side adaptation.
> ...The caching impact of proactive negotiation is far worse than the one extra round trip per site for reactive negotiation, and even that round-trip isn't necessary in formats that support client-side adaptation.
On the caching impact, Simon Willison wrote:
> ...you can’t deploy an application that uses content negotiation via the Accept header behind the Cloudflare CDN — for example serving JSON or HTML for the same URL depending on the incoming Accept header. If you do, Cloudflare may serve cached JSON to an HTML client or vice-versa.
Note: I posted this in another comment with links to those two quotes which I couldn't copy easily now - will add later.
chuckadams 2 days ago [-]
If you do content negotiation, then it’s imperative to send “Vary: accept” in your response. CF and all other CDNs will automatically do the right thing when they see that header.
Content negotiation still has its uses, but most of the time you’re better off using different endpoints.
stillpointlab 1 days ago [-]
Many CDNs do not support arbitrary Vary headers. Cloudfront, for example asks you to create a "Cache and Origin Request Policies" that gives you the ability to choose what headers are part of the request that gets sent to your origin. These will get added to the cache key, but it is a static list based on the request to the origin and not the response.
Akamai is another case where Vary is harmful, from their docs [1]:
> As the content in response may be different for the same URL,
> Akamai edge servers don't cache responses that include the Vary header,
> even if the content is cacheable by definition. The only exception is
> the case where the Vary header's value is Accept-Encoding and the
> Content-Encoding header's value is br or gzip – edge servers cache
> such responses, applying the caching rules you defined in your property.
Cloudflare's docs do seem to indicate they support the Vary header as does Fastly. But one should read the docs of their CDN to find out the behavior. Do not assume Vary is supported.
It's often better to implement your caching logic at the Edge than just going with Vary support. Vanilla Vary support leads to cache dilution.
Accept (and other headers you might include in vary) may take many different values that you want to tie to a single cache entry.
9dev 2 days ago [-]
Conceptually, I’m not sure I agree. There’s elegance in clients saying "I want this resource, and I’d like to get your markdown version of it. If you don’t have one, I’ll also take HTML." And if you couple that with optional "file extensions" at the end of the url to force a specific format (say, /foo for automatic negotiation, and /foo.html, /foo.json, or /foo.md for the respective media type,) you have a very easy to use API that adapts to the client; not the other way around.
I take the point that it makes caching harder, but I don’t think that should overrule ergonomics concerns.
k1m 1 days ago [-]
In general I think I just don't like the idea of one URL being able to return different content. Forces me to think about what each system I give that URL to may be sending in content negotiation headers. Would rather the HTML is returned and alternatives listed in HTML head.
But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know why anyone interested in this wouldn't just improve their HTML markup and leave it to the agent to do the rest. Convert to markdown or extract the useful HTML before handing it to model.
_heimdall 1 days ago [-]
The ideas is that the URL references the resource and the content type requested is only asking for that content in a different projection or representation.
The content at a URL should always match, the format in which its represented can be different based on the request. Its a bit like buying a book in hard copy or paperback, same book different format.
spider-mario 1 days ago [-]
It also means you can’t easily know the set of formats that the server could respond with, which in turns makes e.g. archival more difficult.
9dev 1 days ago [-]
That entirely depends on the server. A good solution would be to include Link headers in all responses:
For the LLM use, the challenge is that it will only discover those after first requesting and parsing the HTML version.
Maybe it will notice those, and maybe it will figure out the pattern for follow-up page requests, but there's no guarantee and it won't help the first request.
9dev 1 days ago [-]
Not necessarily. They could also send a HEAD request to the URL first, to see the headers only and decide on the available alternates.
I am well aware that few sites are taking that much care of their API in terms of HTTP features, but all of the problems discussed here have solid and battle-tested answers.
_heimdall 7 hours ago [-]
The parent was referring to adding `link` tags in the `<head>`, not in HTTP headers.
9dev 6 hours ago [-]
I realised that after posting, but then I suggested sending them as header in the first place.
iamflimflam1 1 days ago [-]
That only works if the client looks at it. The current Claude fetch system does not.
9dev 1 days ago [-]
Should we let vibe-coded agent harnesses dictate protocol design now..?
On the flip side, I'd argue that the current centralisation of user agents (in the classical sense here) that benefit from programmatic content negotiation in form of a handful of harnesses like Claude or Codex is a great lever toward forcing the ecosystem to adopt better practices: If Anthropic added content negotiation as described in this thread to Claude, many sites would be incentivised to improve their web servers.
troupo 1 days ago [-]
> I just don't like the idea of one URL being able to return different content.
It's different content representations. A text in a markdown file is conceptually the same content as the same text in HTML (or PDF).
But it's something I think developers should think about if they rely on caching. If it took Cloudflare this long to support this, there may be other systems which still don't.
Wait, are you telling me that, until two months ago, Cloudflare would actively remove the security from a site that processed Sec-Fetch-* and correctly set the Vary header?
Seriously?
zzo38computer 1 days ago [-]
> It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know which image format would be usable on all UAs and there was no CSS or javascript to allow for client-side adaptation.
Doing this with the Accept header is a bad idea, although I think CSS and JavaScripts (in web pages) is not a good solution to this either (they can often make it worse).
My way is the Scorpion conversion file, which must be downloaded explicitly by the end user and the end user must be allowed to override it with their own, and which tells it what to do when it receives a file that it does not recognize, based on the URL or the file type, such as: rewrite the URL, use a uxn program to convert it (to a format that you can use), use a uxn program to display it, etc. Something similar might be possible to add into WWW, by adding a "Interpreter:" response header into HTTP, perhaps using WebAssembly instead of uxn.
singpolyma3 2 days ago [-]
If CloudFlare isn't honouring vary: accept that's a pretty serious bug
chuckadams 1 days ago [-]
They just started fully honoring it last month. I guess my comment above was overly optimistic.
lo_fye 1 days ago [-]
Please don’t tell me we are willing to make it easier for AI to read our content than we are to make it easier for humans to read our content.
Better idea: make your UI always prioritize the content.
ActionHank 23 hours ago [-]
I think you misunderstand how this plays out.
It's more along the lines of "Oh, hey there, you look like a bot, here are details on how to use our monetized mcp".
More likely though is that nothing will change here, because people will abandon it as soon as there is a paywall.
2000UltraDeluxe 24 hours ago [-]
Don't point it out, please. I, for one, look forward to accessing content without all the bloat.
lekevicius 2 days ago [-]
I'll do that once any of the top 4 AI chatbots says they'll start making requests with this header. Before that it's just a neat idea with no adoption.
I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
JimDabell 1 days ago [-]
Claude Code’s been doing this for at least a year already and I assume Claude Cowork does it too as a result. That’s already millions of users, so it’s not “just a neat idea with no adoption”.
vidarh 1 days ago [-]
Anthropic's own documentation site also returns markdown (of sorts) if you append ".md", and Claude Code knows it.
Also it's not that hard at crawl time to transform HTML into markdown before passing to the LLM.
kennywinker 2 days ago [-]
I mean, it's exceptionally easy for them to add it. Even with low adoption. Somewhere in the bowels of chatgpt there is code that queries for html, and then parses that to markdown or something similar. All this would be is adding the logic to request markdown if available, then bypass the html->markdown step if you get markdown back.
collimarco 2 days ago [-]
What about clean, semantic HTML?
It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now?
There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.
meindnoch 2 days ago [-]
>What about clean, semantic HTML?
Which React package is this?
1 days ago [-]
k1m 1 days ago [-]
I agree. I think a lot of people here are assuming that the full HTML retrieved has to go into the LLM eating up tokens. But why wouldn't the agent try to clean up first and remove bloat and convert to markdown itself, before feeding into LLM. Semantic HTML would make that easier.
troupo 1 days ago [-]
> But why wouldn't the agent try to clean up first and remove bloat and convert to markdown itself, before feeding into LLM.
There's no "agent". It's a few wrappers around API calls in a trenchcoat.
slowin 2 days ago [-]
Presumably markdown uses far fewer tokens.
dubcanada 1 days ago [-]
Then convert from HTML to markdown before you convert to tokens? It's not rocket science, if markdown is "better" stripping all possible HTML tags and leaving just text, images also works. That is more then likely what any automated "serve markdown on the fly" would end up doing.
honr 2 days ago [-]
Is that even true? I most often use HTML. HTML is about 5%-20% more tokens than a similar Markdown. As a rule of thumb, the number of tags/structural tokens doubles, when going from markdown to html, while the rest don't change much. On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.
SyneRyder 1 days ago [-]
> HTML is about 5%-20% more tokens than a similar Markdown.
I'm not seeing this at all. I've got a small search engine I made that strips HTML back to Markdown for its full-text indexing. HTML is typically 10x bigger than the Markdown of the actual content, but that's because the majority of HTML out there is truly terrible.
I personally like HTML, and my own webpages are all hand-coded HTML. In that case, it's probably a closer ratio to what you describe. I'd suggest it's much higher than 20% more, but it's not likely more than double. But that's assuming someone paying attention to the efficiency of the HTML, and most people / websites just don't.
Markdown is even more readable without tools than HTML - it's essentially a plain text document - but I agree that HTML is better for actual semantic structure.
honr 21 hours ago [-]
We are comparing different things.
I am comparing expressing information in semantic HTML or in Markdown. HTML doesn't bloat it much, and it is easier to work with (programmatically in python, ts, go, java, etc.; including parsing and composing and formatting). I am not talking about web apps, which commonly have more "app" than "information".
And the ratio is really closer to 1.1x tokens than 1.2x, unless the information you are expressing is highly structured. Typically 1 structural element (html tag, markdown marks such as ###) ends up being each 1 token. HTML often has the matching closing tag, and that is really what the difference in number of tokens boils down to. Feel free to run your own tests and compare (I ran a few and it was within 1.09-1.14x).
SyneRyder 19 hours ago [-]
I'll give a concrete example, but I think we might be talking at cross-purposes. A reasonably randomly chosen news article:
This HTML page is 360523 bytes of raw HTML. After it has gone through my Markdown parser, it is just 9915 bytes of plain text / Markdown, about a 97% reduction. Only 3% of the HTML is actual content.
The point of Accept Markdown is to save web hosts bandwidth. An AI harness can (and already does) download the HTML & parses out a Markdown version so that it is only minimal tokens before it hits the context window. But I still need to download the 360KB of HTML from the server in order to extract the 9KB I actually need. By serving Markdown versions of your page to AI agents, you can save 97% of the bandwidth that AI agents might be incurring.
There's no reason this page needs to be 360KB of HTML. A handcoded / handoptimized HTML file might be only 12KB - converting from Markdown to HTML is only going to minimal file increase. But that isn't what the web is. It's full of slop generated by CMS applications & Bootstrap & web frameworks and relics of Frontpage edited WYSIWYG HTML editors. Accept Markdown is trying to get webhosts to save everyone bandwidth by serving the Markdown from their side. Maybe it has a chance if it gets baked in at the web server level, or because Cloudflare is applying it to sites that flow through their network. But I think it's ultimately futile - the same people who don't know their Wordpress output is garbage, also won't know how to configure Markdown on their server.
honr 14 hours ago [-]
I am still not getting the point or we are talking past each other.
When I am providing information for humans, I CAN go with semantic HTML (that 9kB Markdown becoming maybe 10KB semantic HTML). I can also go with "framework" layers and turn that into a massive 1MB crap which when rendered by the end user's browser maybe shows most of my content plus ads plus supply chain attacks I unintentionally pulled through with one of the "framework" layers, ahem, nodeJS, ahem.
When I am providing data for both humans and AI agents, XML and other formal data formats might be good choices. If the data happens to be mostly text, then HTML and Markdown both work, but HTML is easier to compose (with programs). If you don't need composition at all, markdown or plain text are obviously easier to hand-craft.
Only in the last scenario, where you don't need to combine information and are directly writing snippets of text, Markdown comes out as a decent option. I honestly don't have a clue how often that happens in practice. Maybe for casual writers (which may be a big market that I am blind to)? As soon as I want to put in a plain table in the data, Markdown gets in the way.
slowin 2 days ago [-]
> On the other hand, I can view HTML without any extra/unusual tools. And composing HTML when I need a bit of structure is far easier than composing markdown.
This is kind of the opposite of reality no? Markdown is just plain text and meant to be human readable. You don't need XML tags to read and write it, opposed to html where you do and you need a browser to properly view it.
honr 1 days ago [-]
No, it's just that I wasn't very clear.
HTML I can view in any browser / webview / etc. Good markdown viewers are fewer / more special, or end up translating md to html for display.
And by composing, I didn't mean writing by hand. We are talking about prompting, right? Or that is what I thought we are talking about. Composing HTML "components" into a final prompt HTML is easier than composing markdown snippets into the final prompt. That is because with HTML there are several ergonomic libraries to parse HTML to AST and to format AST back to HTML. The libraries (for parsing to AST and back to strings) are more limited with markdown.
sebastiennight 21 hours ago [-]
A "prompt" is what goes into the LLM, so I'm not sure what you mean by "a final prompt HTML".
honr 14 hours ago [-]
I often compose prompts from various sources (my "AGENTS.md", "CURRENT_TASK", "CURRENT_PHASE", ...). Last year I was composing in markdown and that got very tedious very quickly. I tried various text and text-like formats, and it turned out HTML is already one of the best formats for this, including for an "AGENTS.md" (which I keep in HTML despite the required ".md" extension).
Of course, if you build up your prompt entirely by hand (copy pasting snippets, etc.) you can get away with markdown or plain text. That works okay for some harnesses, but it leaves too much control (or more accurately, opportunity to misunderstand) to the harness.
xienze 2 days ago [-]
That's highly dependent on what sites you're visiting. Take a look around at a lot of modern sites, there's a sea of divs and spans. Markdown conversion helps LLMs a lot.
honr 1 days ago [-]
Aah, I thought we are talking about prompting or providing information to AI agents in either html or md form, and comparing the two.
Assuming that is what we are talking about, HTML is easier to work with than Markdown, unless you are writing it by hand. That is, composing semantic HTML is more ergonomic than composing a Markdown formatted document from components / snippets, programmatically. The libraries are just better and more versatile in most programming languages. Typically you go from HTML or Markdown to AST, then you compose them to end up with the final tree, then you format the tree to HTML or Markdown. LLMs treat them basically identically (context in HTML or context in Markdown), so I have ended up forming complex prompts / context parts using HTML.
simonw 2 days ago [-]
That used to matter to me back in the days when the best models still only accepted ~32,000 tokens, but these days even the models that run on my laptop are happy with ~100,000 and the hosted models I use take ~200,000 or more.
selcuka 2 days ago [-]
They accept more tokens these days, but they are still more accurate with a shorter context [1].
If it's one of many tool calls, I'd assume that less is more.
simonw 1 days ago [-]
The trick there is to use a subagent to read the HTML page and extract the relevant information, than dumping all that HTML into your top-level session.
That's effectively using an LLM as an HTML to markdown converter, which is both absurdly wasteful and also surprisingly inexpensive (if you use a model like GPT-5.6 Luna.)
honr 21 hours ago [-]
I found HTML itself works [slightly] better with some LLMs (that I happen to frequent). So, when scraping, I parse the resulting html, simplify it, and produce simpler html contents (closer to semantics of what I guessed the real content was). Given processing tools for html are far more mature, I tend to keep content in semantic, low structure html.
o_m 1 days ago [-]
Markdown isn't as expressive. Not all HTML content can be converted to Markdown without losing some of the semantics
Zardoz84 1 days ago [-]
Not my problem. Currently my problem is the high traffic of bots that kills our web apps and don't respect robots.txt or meta tags . We are currently deploying Anubis.
joshum97 2 days ago [-]
I feel like I am going insane. Who in their right mind would feed an LLM raw HTML in the first place??
HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.
We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.
alsetmusic 2 days ago [-]
> in many cases I’d love to skip the CSS/JS and read the markdown directly
I've been enjoying https://defuddle.md since learning about it. Works great. Not affiliated.
usef- 2 days ago [-]
Exa also has an API for it that has worked well for me, returning markdown for a URL, which means you don't need to render js or anything yourself. It doesn't need an account for up to 1k requests/month, which is more than I've ever needed.
Agree. I think many people forget that not long ago, HTML markup on many sites was a lot richer than it is today. Making it trivial to produce a good trimmed down markdown version.
The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily.
I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.
pjerem 1 days ago [-]
> Who in their right mind would feed an LLM raw HTML in the first place??
I'd even say, is that even a problem ? If your website isn't a front end blob soup and just a relatively clean HTML file, your LLM is totally capable to handle it anyway.
And I'm still wondering myself which incentive do I have to serve my content to LLMs anyway.
joshum97 20 hours ago [-]
There's nothing wrong with "dirty" HTML. Modern layouts require lots of wrapper elements--certainly some sites use them unnecessarily, but it's not reasonable to expect a nice modern layout to use the same number of elements as a basic document. And utility CSS has merits. That's a totally separate concern from inaccessible div soup--you can still use sections, navs, aria- attributes, etc.
If you're working with a marketing team, you'll find that they are desperate to serve their content to LLMs, to counteract the rapidly shrinking number of people actually visiting websites. What I don't want is for that to mean "build the website twice, one for humans and one for LLMs". Websites are for people—bots can adapt.
tigrandza 1 days ago [-]
i have built 2 html to document pipelines and read it back with pdftotext, and I can say that "just convert it yourself" step is the place where things are getting wrong.
For example Markdown has no support for the subscript and superscript in CommonMark or GFM and if you have <sup>4</sup><sub>3</sub> becomes 43 and water formula becomes H2O so in the output you get not the ugly but the wrong one. And no converter can fix it, because that's a target format which can't express it. extractors like Defuddle don't save from this also. and non of those extractors or converter throws you just get wrong text
Semiapies 1 days ago [-]
Who in their right mind would feed any random LLM crawler their website's content?
joshum97 20 hours ago [-]
marketers
0xbadcafebee 1 days ago [-]
I think you are operating under the misconception that software developers are in their right mind
> Let's say you want to plan a trip to Thailand with your family. You could use the fancy AI to do it for you, or you could build a stupid frontend with minimal natural language understanding.
Except it isn't. Someone else could build that stupid frontend, and essentially create a better booking.com. Or you could ask the robot to make you a stupid frontend. It remains true that a well-designed MCP server goes wildly against the interests of sites like booking.com.
phoghed 24 hours ago [-]
Yes, almost every website on earth is enshittified to the point that it’s user hostile.
Even something as simple as a recipe blog. That one is already solved by chatbots, and it’s glorious. I never have to look at a shitty recipe site full of ads with the recipe buried beneath a mile of fake stories again. LLM finds some, reads reviews, I pick one, if it’s good import to paprika, and that’s that.
There’s at least a chance that enough people demand it, combined with AI hype, that we could end up with a slightly better experience.
And if not, then there’s always the brute force approach of something like grok bot
qznc 2 days ago [-]
Yeah, that is why this will not get popular.
jiehong 2 days ago [-]
Markdown ads?
qingcharles 2 days ago [-]
Time magazine already serves their pages like this to agents with ads for the agents in them, IIRC.
blitzar 1 days ago [-]
Whats the sales agents conversion rate of selling agents to agents like?
LunicLynx 2 days ago [-]
Hello … AI company that wants easier access to data
MiroslavPokorny 1 days ago [-]
Why would any website want to put extra effort to "give" their content as markdown to AI and get nothing in return ?
gampleman 1 days ago [-]
An obvious example is if my product has an SDK and I'd like my customers to be able to script my product via their LLMs then I want to make my docs as AI-friendly as possible, since if their LLM picks my product's SDK rather than the competition, I make $$$.
Semiapies 1 days ago [-]
A question repeatedly asked by different people throughout these comments, but nobody answers.
maurelius2 2 days ago [-]
Even without AI this would be a nice feature.
Kuyawa 2 days ago [-]
Interesting...
A cherry.jpg image/jpeg file is shown as an image by the browser
So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki
Browsers should implement that asap and we will follow
meindnoch 2 days ago [-]
It depends on one question: does Markdown support <script> tags?
krapp 2 days ago [-]
Technically, Markdown supports all HTML tags, including script tags.
Practically, Markdown is more of a vibe than a spec and everyone just uses whatever subset makes sense to them.
zzo38computer 1 days ago [-]
The web page does not load for me (I cannot reach the server), but the idea to serve Markdown according to the Accept header is possible, although it might be better to do that by changing the file name and adding a header (or a <meta> command) to indicate this possibility, so that there is a different file name in each case. Some already do this, allowing to change the file name for JSON or Markdown instead of HTML, and is helpful (there are also other common ways that you can change a URL to access it differently e.g. "blob" to "raw" in a git viewer, and I use these things sometimes). However, the reason should not be mainly for AI agents, even if it can be helpful for that purpose too.
Dwedit 1 days ago [-]
How about allowing browsers to actually display markdown?
ilikejam 23 hours ago [-]
Still slightly amazing that browsers don't render markdown natively.
giorgioz 23 hours ago [-]
Okay but than which agent will actually use that? Because if no agent is going to use the accept html header to get markdown this is pointless.
arjie 2 days ago [-]
Everything is determined by cloudflare. If they supported Gemini on the same hostname I’d serve it everywhere. If they supported text/plain and text/html everywhere I’d serve that. But they don’t cache per content encoding so I just do what works for them.
Quite helpful, I’ll see if I can serve text/plain.
rbanffy 1 days ago [-]
Sounds like a nice idea for text-based browsers. Also, it makes a lot of advertising difficult.
OTOH, serving ads to mistrain AI content is an interesting business model.
monneyboi 2 days ago [-]
Yeah, this is the way forward. All the POW stuff is such a waste of energy.
The open internet deserves a real solution instead of gatekeepers in the form of brightdata, firecrawl, cloudflare and the likes.
brap 2 days ago [-]
Best case scenario, this ends up being abused in order to feed LLMs crap responses (or worse).
fergie 1 days ago [-]
Unless I'm missing something here, this is all wrong. Markdown is OK, but for a variety of reasons, HTML is much better.
What you _should_ be doing is ensuring that your page works with screen readers. That simultaneously solves any kind of machine-readability issues and also makes your website accessible for the blind.
nozzlegear 2 days ago [-]
No, I don't think I will. I publish things for people, not bots.
4lb0 2 days ago [-]
Is this an alternative or complements llms.txt?
kimseungyong 1 days ago [-]
[dead]
a2ff6eeb0 2 days ago [-]
Hm, interesting avenue for prompt injection.
spockz 2 days ago [-]
How is that different from today where agents just do searches and ingest web pages?
hnlmorg 2 days ago [-]
I don’t disagree with you in principle but there is a difference.
The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.
spockz 2 days ago [-]
Right. And today we have something similar with search engines going on where sites want to offer the full text to the engine for seo. But maybe they want to do the opposite to agent and poison the well. So human agents still get the “real” data and agents the “something plausible but not quite right” data.
Or everything just disappears behind something like cloudflare or a paywall.
hnlmorg 1 days ago [-]
Funny enough, I was thinking about the SEO too when I wrote my comment.
The problem is that it’s in the sites interest to be SEOed. But it’s against their interests to be AI crawled.
spockz 20 hours ago [-]
If AI agents would attribute earnings to sites that contributed to the right answer we have completed the circle again. At least then it could become useful to serve agents.
hnlmorg 12 hours ago [-]
That would be nice but I feel that’s even less likely to happen than this markdown proposal.
OutOfHere 1 days ago [-]
It's a good thought, but the problem with it is that it can never be trusted by the bot, except for a finite whitelisted set of trusted sites that are known to serve representative markdown that faithfully represents the HTML page, its content, and its links.
2 days ago [-]
ErroneousBosh 2 days ago [-]
So can I use this to serve actively harmful content to the Cocaine Piracy Parrots?
Maybe tarpit them into reading gigabytes of Markov Chain nonsense, really slowly?
xgulfie 2 days ago [-]
Sounds like a great way to spread disinfo to LLMs, so I'm down
akomtu 1 days ago [-]
This will be used to serve SEO slop. LLMs are already used as search engines and businesses really want to outrank competition by any means possible.
> It is a bad design trade-off to send a bunch of header fields on every request just to tell the server all of the possible variations of preference held by the user, particularly when there is a very small chance that any of those dimensions are applicable to the target resource. It has been a bad design trade-off ever since the very brief period in 1993-94 when folks didn't know which image format would be usable on all UAs and there was no CSS or javascript to allow for client-side adaptation.
> ...The caching impact of proactive negotiation is far worse than the one extra round trip per site for reactive negotiation, and even that round-trip isn't necessary in formats that support client-side adaptation.
On the caching impact, Simon Willison wrote:
> ...you can’t deploy an application that uses content negotiation via the Accept header behind the Cloudflare CDN — for example serving JSON or HTML for the same URL depending on the incoming Accept header. If you do, Cloudflare may serve cached JSON to an HTML client or vice-versa.
Note: I posted this in another comment with links to those two quotes which I couldn't copy easily now - will add later.
Content negotiation still has its uses, but most of the time you’re better off using different endpoints.
Akamai is another case where Vary is harmful, from their docs [1]:
Cloudflare's docs do seem to indicate they support the Vary header as does Fastly. But one should read the docs of their CDN to find out the behavior. Do not assume Vary is supported.https://techdocs.akamai.com/property-mgr/docs/rm-vary-header
I take the point that it makes caching harder, but I don’t think that should overrule ergonomics concerns.
But for HTML and Markdown in particular, there's been so much useful work done in the semantic HTML space and microformats, that I don't know why anyone interested in this wouldn't just improve their HTML markup and leave it to the agent to do the rest. Convert to markdown or extract the useful HTML before handing it to model.
The content at a URL should always match, the format in which its represented can be different based on the request. Its a bit like buying a book in hard copy or paperback, same book different format.
Maybe it will notice those, and maybe it will figure out the pattern for follow-up page requests, but there's no guarantee and it won't help the first request.
I am well aware that few sites are taking that much care of their API in terms of HTTP features, but all of the problems discussed here have solid and battle-tested answers.
On the flip side, I'd argue that the current centralisation of user agents (in the classical sense here) that benefit from programmatic content negotiation in form of a handful of harnesses like Claude or Codex is a great lever toward forcing the ecosystem to adopt better practices: If Anthropic added content negotiation as described in this thread to Claude, many sites would be incentivised to improve their web servers.
It's different content representations. A text in a markdown file is conceptually the same content as the same text in HTML (or PDF).
Wrong: https://developers.cloudflare.com/cache/concepts/vary/
Simon wrote that in 2023: https://simonwillison.net/2023/Nov/20/cloudflare-does-not-co...
But it's something I think developers should think about if they rely on caching. If it took Cloudflare this long to support this, there may be other systems which still don't.
Link to Roy Fielding comment:
https://lists.w3.org/Archives/Public/ietf-http-wg/2013JanMar...
Seriously?
Doing this with the Accept header is a bad idea, although I think CSS and JavaScripts (in web pages) is not a good solution to this either (they can often make it worse).
My way is the Scorpion conversion file, which must be downloaded explicitly by the end user and the end user must be allowed to override it with their own, and which tells it what to do when it receives a file that it does not recognize, based on the URL or the file type, such as: rewrite the URL, use a uxn program to convert it (to a format that you can use), use a uxn program to display it, etc. Something similar might be possible to add into WWW, by adding a "Interpreter:" response header into HTTP, perhaps using WebAssembly instead of uxn.
Better idea: make your UI always prioritize the content.
It's more along the lines of "Oh, hey there, you look like a bot, here are details on how to use our monetized mcp".
More likely though is that nothing will change here, because people will abandon it as soon as there is a paywall.
I also think it's exceedingly unlikely that any of the top 4 chatbots would choose to load websites this way. Too many risks for a 0.01% adoption even years later.
E.g.: https://code.claude.com/docs/en/overview.md
https://acceptmarkdown.com/status
It was already optimized for bots and search engines (which are bots) and it has been used for decades. Why we need to serve in markdown now?
There are also many parts of the HTML, like navs, that are useful for bots and AI and may be removed in the markdown version.
Which React package is this?
There's no "agent". It's a few wrappers around API calls in a trenchcoat.
I'm not seeing this at all. I've got a small search engine I made that strips HTML back to Markdown for its full-text indexing. HTML is typically 10x bigger than the Markdown of the actual content, but that's because the majority of HTML out there is truly terrible.
I personally like HTML, and my own webpages are all hand-coded HTML. In that case, it's probably a closer ratio to what you describe. I'd suggest it's much higher than 20% more, but it's not likely more than double. But that's assuming someone paying attention to the efficiency of the HTML, and most people / websites just don't.
Markdown is even more readable without tools than HTML - it's essentially a plain text document - but I agree that HTML is better for actual semantic structure.
I am comparing expressing information in semantic HTML or in Markdown. HTML doesn't bloat it much, and it is easier to work with (programmatically in python, ts, go, java, etc.; including parsing and composing and formatting). I am not talking about web apps, which commonly have more "app" than "information".
And the ratio is really closer to 1.1x tokens than 1.2x, unless the information you are expressing is highly structured. Typically 1 structural element (html tag, markdown marks such as ###) ends up being each 1 token. HTML often has the matching closing tag, and that is really what the difference in number of tokens boils down to. Feel free to run your own tests and compare (I ran a few and it was within 1.09-1.14x).
https://www.abc.net.au/news/2026-08-27/shania-twain-intervie...
This HTML page is 360523 bytes of raw HTML. After it has gone through my Markdown parser, it is just 9915 bytes of plain text / Markdown, about a 97% reduction. Only 3% of the HTML is actual content.
The point of Accept Markdown is to save web hosts bandwidth. An AI harness can (and already does) download the HTML & parses out a Markdown version so that it is only minimal tokens before it hits the context window. But I still need to download the 360KB of HTML from the server in order to extract the 9KB I actually need. By serving Markdown versions of your page to AI agents, you can save 97% of the bandwidth that AI agents might be incurring.
There's no reason this page needs to be 360KB of HTML. A handcoded / handoptimized HTML file might be only 12KB - converting from Markdown to HTML is only going to minimal file increase. But that isn't what the web is. It's full of slop generated by CMS applications & Bootstrap & web frameworks and relics of Frontpage edited WYSIWYG HTML editors. Accept Markdown is trying to get webhosts to save everyone bandwidth by serving the Markdown from their side. Maybe it has a chance if it gets baked in at the web server level, or because Cloudflare is applying it to sites that flow through their network. But I think it's ultimately futile - the same people who don't know their Wordpress output is garbage, also won't know how to configure Markdown on their server.
When I am providing information for humans, I CAN go with semantic HTML (that 9kB Markdown becoming maybe 10KB semantic HTML). I can also go with "framework" layers and turn that into a massive 1MB crap which when rendered by the end user's browser maybe shows most of my content plus ads plus supply chain attacks I unintentionally pulled through with one of the "framework" layers, ahem, nodeJS, ahem.
When I am providing data for both humans and AI agents, XML and other formal data formats might be good choices. If the data happens to be mostly text, then HTML and Markdown both work, but HTML is easier to compose (with programs). If you don't need composition at all, markdown or plain text are obviously easier to hand-craft.
Only in the last scenario, where you don't need to combine information and are directly writing snippets of text, Markdown comes out as a decent option. I honestly don't have a clue how often that happens in practice. Maybe for casual writers (which may be a big market that I am blind to)? As soon as I want to put in a plain table in the data, Markdown gets in the way.
This is kind of the opposite of reality no? Markdown is just plain text and meant to be human readable. You don't need XML tags to read and write it, opposed to html where you do and you need a browser to properly view it.
HTML I can view in any browser / webview / etc. Good markdown viewers are fewer / more special, or end up translating md to html for display.
And by composing, I didn't mean writing by hand. We are talking about prompting, right? Or that is what I thought we are talking about. Composing HTML "components" into a final prompt HTML is easier than composing markdown snippets into the final prompt. That is because with HTML there are several ergonomic libraries to parse HTML to AST and to format AST back to HTML. The libraries (for parsing to AST and back to strings) are more limited with markdown.
Of course, if you build up your prompt entirely by hand (copy pasting snippets, etc.) you can get away with markdown or plain text. That works okay for some harnesses, but it leaves too much control (or more accurately, opportunity to misunderstand) to the harness.
Assuming that is what we are talking about, HTML is easier to work with than Markdown, unless you are writing it by hand. That is, composing semantic HTML is more ergonomic than composing a Markdown formatted document from components / snippets, programmatically. The libraries are just better and more versatile in most programming languages. Typically you go from HTML or Markdown to AST, then you compose them to end up with the final tree, then you format the tree to HTML or Markdown. LLMs treat them basically identically (context in HTML or context in Markdown), so I have ended up forming complex prompts / context parts using HTML.
[1] https://arxiv.org/abs/2307.03172
That's effectively using an LLM as an HTML to markdown converter, which is both absurdly wasteful and also surprisingly inexpensive (if you use a model like GPT-5.6 Luna.)
HTML is a markup language. User agents present it in a way that makes sense for the user—visually, or through assistive technology. Nothing about adding LLM “users” should change this—their user agent, the harness, should present HTML in a way they can natively understand, by translating it to Markdown.
We are not going to rewrite the entire web because harness developers are too lazy or stupid to pull an HTML to markdown package from npm. If some sites want to do so anyway, good for them, in many cases I’d love to skip the CSS/JS and read the markdown directly (or better, nicely formatted). But don’t blame website authors for your harness wasting your tokens.
I've been enjoying https://defuddle.md since learning about it. Works great. Not affiliated.
Most harnesses have a "zero config" plugin that works, eg: https://pi.dev/packages/pi-exa
(no affiliation to either)
The reason it may be more difficult today is because we've lost a lot of that. Some of it because of modern JS frameworks, but some also because publishers simply don't want to make it easy for the useful stuff to be scraped and extracted easily.
I'm not convinced that's changing because of AI agents (it's getting worse in many ways with anti-agent rules). Maybe improving for documentation pages intended for agents. But if it is changing, I think it'd be far easier to improve the HTML and let the agent take care of the rest.
I'd even say, is that even a problem ? If your website isn't a front end blob soup and just a relatively clean HTML file, your LLM is totally capable to handle it anyway.
And I'm still wondering myself which incentive do I have to serve my content to LLMs anyway.
If you're working with a marketing team, you'll find that they are desperate to serve their content to LLMs, to counteract the rapidly shrinking number of people actually visiting websites. What I don't want is for that to mean "build the website twice, one for humans and one for LLMs". Websites are for people—bots can adapt.
> Why do you need AI then?
(from this HN comment, worth a read in full: https://news.ycombinator.com/item?id=43679585)
It's giving https://news.ycombinator.com/item?id=8863#9224
Except it isn't. Someone else could build that stupid frontend, and essentially create a better booking.com. Or you could ask the robot to make you a stupid frontend. It remains true that a well-designed MCP server goes wildly against the interests of sites like booking.com.
Even something as simple as a recipe blog. That one is already solved by chatbots, and it’s glorious. I never have to look at a shitty recipe site full of ads with the recipe buried beneath a mile of fake stories again. LLM finds some, reads reviews, I pick one, if it’s good import to paprika, and that’s that.
There’s at least a chance that enough people demand it, combined with AI hype, that we could end up with a slightly better experience.
And if not, then there’s always the brute force approach of something like grok bot
A cherry.jpg image/jpeg file is shown as an image by the browser
A logo in svg like https://news.ycombinator.com/y18.svg too
So browsers should show readme.md as text/markdown and clicking on a link inside that file should also redirect to another markdown file, instant wiki
Browsers should implement that asap and we will follow
Practically, Markdown is more of a vibe than a spec and everyone just uses whatever subset makes sense to them.
They’re far too useful for me.
OTOH, serving ads to mistrain AI content is an interesting business model.
The open internet deserves a real solution instead of gatekeepers in the form of brightdata, firecrawl, cloudflare and the likes.
What you _should_ be doing is ensuring that your page works with screen readers. That simultaneously solves any kind of machine-readability issues and also makes your website accessible for the blind.
The MD content isn’t intended for human consumption whereas HTML is. So you either have that injection readable to your users, which could be jarring to non-technical readers, or you play the cat-and-mouse game of hoping those pesky crawlers don’t ignore text that’s not human-visible on a rendered page.
Or everything just disappears behind something like cloudflare or a paywall.
The problem is that it’s in the sites interest to be SEOed. But it’s against their interests to be AI crawled.
Maybe tarpit them into reading gigabytes of Markov Chain nonsense, really slowly?