It's strange to me that LLMs seem to motivate this, instead of "your source artifacts should live with your code..."
...and source artifacts include decision records of various sorts, lab notebooks used to run experiments, etc. Yeah, someone trying to understand the code needs access to all those; make it accessible to them.
I believe that agents should not be used to generate much content in /src/md. This directory should be mainly human authored and curated.
I think that’s the key to this proposal but it’s buried deep down in the article. Otherwise people would assume plan mode slop which I find is excruciating to review.
I don't know how this is actually supposed to work. I do write detailed initial prompts, and I often try to capture those (usually in the commit message), but for anything nontrivial there follows several back and forth rounds where I'm telling it "actually, rip out the Foo and try just augmenting Bar" or "add some tests for condition X (and fix the code if they fail)" or whatever. (And no, I don't think this will go away with better models, because often I don't know what I want without trying something and seeing how it looks.) Even if I was capturing these later prompts, they aren't very easy to interpret in the absence of the earlier iterations of the code, which I'm throwing away. I could capture the whole session, and I've seen people make the case for that, but that's not quite what's being asked for here.
I think a possible consequence of the proposed model would be that every time (or every few times) you uncover new things like this, you would have to (manually) incorporate them back into the initial document and restart the session. I would complement with LLM CI reviews that judge whether the final implementation honors what is captured in the docs.
(I haven't tried this, nor am sure if it would be reasonable in practice, but it sounds like an interesting experiment)
In some projects, I will create a root folder docs/. I will then number files YYYY-MM-DD-title.md, or NNN-title.md starting at 000 and counting up. I strictly keep only human-authored documents in this folder. I will do things like spec out interfaces and tests and invariants by hand, and then write automated checks to ensure that the interfaces match what is documented.
I also like to include what I call "Provenance" frontmatter for all files. For human-written markdown/code files, this is something like:
+++
created: YYYY-MM-DD
author: Name <email>
provenance: human
+++
And for AI-generated e.g. code/markdown, this is something like:
// created: YYYY-MM-DD
// model: glorm-9-promax
// driver: Name <email>
// provenance: ai
I have a strict "no AI edits human provenance, no human edits AI provenance" policy; AI edits also "infect" and demote the provenance of human-produced artifacts should I manually break that rule. I will often like to write out e.g. trait, interface definition, or module files with human provenance, then generate a test suite / implementation with AI provenance against that interface definition. I have a standing rule that human provenance always serves as the source of truth.
Really hard pass on this here, but I can definitely see why people want this, it sits right around the "tickets/discussion should be colocated with code" discussions.
Elixir has exdoc, Rust has rustdoc and Unison has {{ fancy doc comments}}, there are also a quite a lot of notebook-style programming environments. I feel like all of those tools have (a lot!) more potential for being integrated with slopped code than seeing markdown as source code in a directory.
It took years for type systems(ergonomic ones like OCaml/Elm) to catch some traction, and now we want to go and write some unchecked Markdown and see it as part of a system? Am I out of touch, or is that really the way to make resilient systems?
I’m not getting it either. Type systems are deterministic & are trying to encode invariants in a predictable manner—2 features LLM outputs don’t have. …& this is already frustrating enough that such a bad format for documentation/technical writing, Markdown, is being plopped all over the place; it’s as ergonomic & feature rich as JavaScript’s type system.
This is necessarily the weakest part of this essay because this is a new idea and I haven’t used it extensively yet. It is me thinking out loud and inviting discussion.
I'm glad this was acknowledged because it really makes no sense to have a "markdown" directory with everything in it, just as it doesn't make sense to have a "/src/javascript" file with everything in the same place.
Documentation should live close to what it documents.
bgs_ | 4 hours ago
:c
cceckman | 5 hours ago
It's strange to me that LLMs seem to motivate this, instead of "your source artifacts should live with your code..."
...and source artifacts include decision records of various sorts, lab notebooks used to run experiments, etc. Yeah, someone trying to understand the code needs access to all those; make it accessible to them.
[OP] facundoolano | 7 hours ago
I think that’s the key to this proposal but it’s buried deep down in the article. Otherwise people would assume plan mode slop which I find is excruciating to review.
bakkot | 5 hours ago
I don't know how this is actually supposed to work. I do write detailed initial prompts, and I often try to capture those (usually in the commit message), but for anything nontrivial there follows several back and forth rounds where I'm telling it "actually, rip out the Foo and try just augmenting Bar" or "add some tests for condition X (and fix the code if they fail)" or whatever. (And no, I don't think this will go away with better models, because often I don't know what I want without trying something and seeing how it looks.) Even if I was capturing these later prompts, they aren't very easy to interpret in the absence of the earlier iterations of the code, which I'm throwing away. I could capture the whole session, and I've seen people make the case for that, but that's not quite what's being asked for here.
[OP] facundoolano | 4 hours ago
I think a possible consequence of the proposed model would be that every time (or every few times) you uncover new things like this, you would have to (manually) incorporate them back into the initial document and restart the session. I would complement with LLM CI reviews that judge whether the final implementation honors what is captured in the docs.
(I haven't tried this, nor am sure if it would be reasonable in practice, but it sounds like an interesting experiment)
slightknack | 5 hours ago
In some projects, I will create a root folder
docs/. I will then number filesYYYY-MM-DD-title.md, orNNN-title.mdstarting at000and counting up. I strictly keep only human-authored documents in this folder. I will do things like spec out interfaces and tests and invariants by hand, and then write automated checks to ensure that the interfaces match what is documented.I also like to include what I call "Provenance" frontmatter for all files. For human-written markdown/code files, this is something like:
And for AI-generated e.g. code/markdown, this is something like:
I have a strict "no AI edits human provenance, no human edits AI provenance" policy; AI edits also "infect" and demote the provenance of human-produced artifacts should I manually break that rule. I will often like to write out e.g. trait, interface definition, or module files with human provenance, then generate a test suite / implementation with AI provenance against that interface definition. I have a standing rule that human provenance always serves as the source of truth.
zetashift | 4 hours ago
Really hard pass on this here, but I can definitely see why people want this, it sits right around the "tickets/discussion should be colocated with code" discussions.
Elixir has
exdoc, Rust hasrustdocand Unison has{{ fancy doc comments}}, there are also a quite a lot of notebook-style programming environments. I feel like all of those tools have (a lot!) more potential for being integrated with slopped code than seeing markdown as source code in a directory.It took years for type systems(ergonomic ones like OCaml/Elm) to catch some traction, and now we want to go and write some unchecked Markdown and see it as part of a system? Am I out of touch, or is that really the way to make resilient systems?
toastal | 3 hours ago
I’m not getting it either. Type systems are deterministic & are trying to encode invariants in a predictable manner—2 features LLM outputs don’t have. …& this is already frustrating enough that such a bad format for documentation/technical writing, Markdown, is being plopped all over the place; it’s as ergonomic & feature rich as JavaScript’s type system.
dlisboa | 6 hours ago
I'm glad this was acknowledged because it really makes no sense to have a "markdown" directory with everything in it, just as it doesn't make sense to have a "/src/javascript" file with everything in the same place.
Documentation should live close to what it documents.
cceckman | 5 hours ago
This essay claims:
dlisboa | 5 hours ago
I disagree with that. Either way, we don't categorize source code by file type.
hyperpape | 2 hours ago
Maven's
src/main/javais one example to the contrary, though you're free to argue it's a bad idea.