As Lobsters' token technical writer, I figured the community might be interested to know that among documentarians there's still a lot of debate and unknowns about how agents (and model training pipelines) consume technical documentation.
It was a wonderful read. For at least two reasons. One, it's a beautifully silly way, with the appropriate amount of seriousness and investment, to tell people "to an llm, a file is a file". And another, I've enjoyed reading the article itself, I don't know if an llm or a human generated the text, but it was a nice amusing read with my breakfast.
I'm confused about this whole concept of llms.txt as a SEO thing. It was not the purpose when the idea was created. The usage doesn't depend on the search in any way either. https://llmstxt.org/
As the document says, it's a "file to provide information to help agents use a website". It's not for search. It's for "look at https://docs.convex.dev/llms.txt and tell me how to deploy this application" without the need to burn the tokens on parsing html pages meant for people.
But web pages are built for people. An HTML page wraps its information in navigation, ads, and JavaScript, and converting it back into clean text is difficult and imprecise.
People don’t want this junk either so why save a visual slop version for humans & lose semantics for both humans & machines by going to Markdown. This is just so asinine to me—but also depressing. HTML is already has many semantic elements that go beyond what Markdown can represent & can explains what’s going on + where to go (since screen reader users exist), but HTML can also be extended further with rich formats/microdata. The pages would be lightweight too if visual parts of the page were separate from the markup & not a mountain of 4000 inline style and/or Tailwind classes.
We can even see semantics breaking down right here
# Title
> Optional description goes here
Optional details go here
This isn’t where the optional description goes! > denotes a blockquote in markdown & <blockquote> elements should be quoting a source according to the HTML spec which is what Markdown is usually targeting. HTML I can use Schema.org with a <header><h1 itemprop="headline">…</h1><p itemprop="alternativeHeadline">…</p></header>. With these semantics machines can understand more instead of needing to infer meaning… Folks will tell me the “semantic HTML ship has sailed” & I just have to ask why? If anything this should have been a reason to embrace it instead of needing a bot to go thru more effort to guess intent. …But so much bad HTML has been pushed out that there are services you can buy to turn your bloated website into markdown to optimize for bots.
Engineers are always trying to engineer their way out of problems (often they create) instead of looking at the simple solution & removing the root cause or reach for the old thing that is already good …This is why I’m trying to open a teahouse. This hurts my brain.
They really do (apart from ads). There's a reason we render things rather than serve raw markup. Good hierarchical navigation and interactivity without full reload on every click is welcome.
HTML I can use Schema.org with a <header><h1 itemprop="headline">…</h1><p itemprop="alternativeHeadline">…</p></header>
Yes, but for LLMs you burned 20+ tokens here and on every following header without providing value. Inferring the meaning of the quote-as-alttitle doesn't cost anything and just works. Even stripping html down to basics won't get you close to the efficiency of markdown-like format.
Then I think the machines are being real dumb a schema.org could teach you how to query & reason about the data instead of burning tokens trying to figure out that a > isn’t quoting anything & that there’s basically no semantics to what’s being talked about. Semantic markup also helps non-LLM machines as well render content better for reader modes, querying for scripts/scraping, XSLT, & so on. I think this whole think could have been an XML namespace on HTML.
You need to figure out what to query and do that, so that's multiple turns of tool use to go down a hierarchy of what you need. All of that hits the context use as well. On the other hand there's no "trying to figure out that a > isn’t quoting anything" process and no wasted tokens there - it really doesn't cost you anything. It's more economical to just read badly formatted information in this use case than be fancy with the process.
i think it’s mostly that the “standard” part of it seems meaningless: as you say, it is just a text file that the model can read. the model doesn’t understand or specially support the “standard,” it’s just more text. nothing was stopping people from adding README.txt to their website before, and the value of conforming to the llms.txt format is unclear.
edit: and intention aside, TFA states that they aren’t complaining about the llms.txt thing specifically, they are complaining about the GEO astrology that has come about as a post-hoc justification for using this thing
I had no idea people were trying to use llms.txt for GEO. llms.txt is designed only to make a website easy to view and navigate for agents:
We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content. The file can be placed at the site root, or at any path within it, covering the pages under that path. This file offers brief background information, guidance, and links to detailed markdown files.
[OP] kaycebasques | a day ago
As Lobsters' token technical writer, I figured the community might be interested to know that among documentarians there's still a lot of debate and unknowns about how agents (and model training pipelines) consume technical documentation.
zladuric | 23 hours ago
It was a wonderful read. For at least two reasons. One, it's a beautifully silly way, with the appropriate amount of seriousness and investment, to tell people "to an llm, a file is a file". And another, I've enjoyed reading the article itself, I don't know if an llm or a human generated the text, but it was a nice amusing read with my breakfast.
addison | 14 hours ago
FYI, GEO = "Generative Engine Optimisation", for those like me that did not recognise the acronym.
viraptor | 16 hours ago
I'm confused about this whole concept of llms.txt as a SEO thing. It was not the purpose when the idea was created. The usage doesn't depend on the search in any way either. https://llmstxt.org/
As the document says, it's a "file to provide information to help agents use a website". It's not for search. It's for "look at https://docs.convex.dev/llms.txt and tell me how to deploy this application" without the need to burn the tokens on parsing html pages meant for people.
toastal | 10 hours ago
People don’t want this junk either so why save a visual slop version for humans & lose semantics for both humans & machines by going to Markdown. This is just so asinine to me—but also depressing. HTML is already has many semantic elements that go beyond what Markdown can represent & can explains what’s going on + where to go (since screen reader users exist), but HTML can also be extended further with rich formats/microdata. The pages would be lightweight too if visual parts of the page were separate from the markup & not a mountain of 4000 inline style and/or Tailwind classes.
We can even see semantics breaking down right here
This isn’t where the optional description goes!
>denotes a blockquote in markdown &<blockquote>elements should be quoting a source according to the HTML spec which is what Markdown is usually targeting. HTML I can use Schema.org with a<header><h1 itemprop="headline">…</h1><p itemprop="alternativeHeadline">…</p></header>. With these semantics machines can understand more instead of needing to infer meaning… Folks will tell me the “semantic HTML ship has sailed” & I just have to ask why? If anything this should have been a reason to embrace it instead of needing a bot to go thru more effort to guess intent. …But so much bad HTML has been pushed out that there are services you can buy to turn your bloated website into markdown to optimize for bots.Engineers are always trying to engineer their way out of problems (often they create) instead of looking at the simple solution & removing the root cause or reach for the old thing that is already good …This is why I’m trying to open a teahouse. This hurts my brain.
viraptor | 9 hours ago
They really do (apart from ads). There's a reason we render things rather than serve raw markup. Good hierarchical navigation and interactivity without full reload on every click is welcome.
Yes, but for LLMs you burned 20+ tokens here and on every following header without providing value. Inferring the meaning of the quote-as-alttitle doesn't cost anything and just works. Even stripping html down to basics won't get you close to the efficiency of markdown-like format.
toastal | 7 hours ago
Then I think the machines are being real dumb a schema.org could teach you how to query & reason about the data instead of burning tokens trying to figure out that a
>isn’t quoting anything & that there’s basically no semantics to what’s being talked about. Semantic markup also helps non-LLM machines as well render content better for reader modes, querying for scripts/scraping, XSLT, & so on. I think this whole think could have been an XML namespace on HTML.viraptor | 7 hours ago
You need to figure out what to query and do that, so that's multiple turns of tool use to go down a hierarchy of what you need. All of that hits the context use as well. On the other hand there's no "trying to figure out that a > isn’t quoting anything" process and no wasted tokens there - it really doesn't cost you anything. It's more economical to just read badly formatted information in this use case than be fancy with the process.
sloane | 15 hours ago
i think it’s mostly that the “standard” part of it seems meaningless: as you say, it is just a text file that the model can read. the model doesn’t understand or specially support the “standard,” it’s just more text. nothing was stopping people from adding README.txt to their website before, and the value of conforming to the llms.txt format is unclear.
edit: and intention aside, TFA states that they aren’t complaining about the llms.txt thing specifically, they are complaining about the GEO astrology that has come about as a post-hoc justification for using this thing
ncoop57 | 16 hours ago
I had no idea people were trying to use llms.txt for GEO. llms.txt is designed only to make a website easy to view and navigate for agents:
It was never about search.