Nice post. Good motivation to look more at formal verification. There was that Aristotle post today which seems like a prompt driven, plug and play verifier using lean.
Makes me think that LLM assisted formal requirements generation for a roll your own "Aristotle" is possible.
Just thinking about our own projects in the signal processing domain. A nice path could be: human language -> temporal logic specs for black and white box run-time testing.
There are code generators which will turn formal logic into a run time monitor. In the past, I've never used them past toy examples because its so verbose.
> The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.
That's the difference between an engineer and a script kiddie.
> Good motivation to look more at formal verification.
A good motivation to stray from the conventional (non-)wisdom of using Lean as a solution as well, particularly because it's in many cases worse than much less rigorous methods for correctness, and worse than doing literally nothing special at all.
The really powerful thing you've done here is begin to create a taxonomy of agent patterns. Given we are pretty much all engineers in this space; I am shocked at the lack of engineering taking place. People are picking patterns they find intuitive; for various invented reasons, this is fine in the beginning, but we need to start objectively measuring their performance. It needs to be said that there are an infinite number of plausible agent patterns, and we should view all (from skills to MCP to Audit, whathaveyou) as something that may go the same way as netscape until there is real justification.
I use what I would say is a TDD skill, but I couldn't find anywhere in the article what you meant by TDD? How did you prompt it to use TDD? Why was it less effective? I feel targeted ;)
I'd love to see more of that. I'm currently defaulting to Rust in my software building but F# looks super strong there. On par with Python for correctness with Ultra effort.
I wonder how Rust and other languages are doing in Astra 6/Sol 6.1 and on more projects.
I like research notes, and I like reading what Dan is up to, always. That said, some helpful context to this post - Dan is working on some engineering questions -- how can we best use the tools we have, and how does that evolve?
His general test methods this year have been to provide a specification document of some sort and then prompt a clean room implementation of the spec, which can be tested against holdout tests.
Things move pretty quickly - this summer, his test was Zstd and results were at like 25-30% of tests passing at best. He now considers Zstd saturated, as is IMAP from RFC; pandoc is one that he's used throughout, and is not yet perfect.
Are my eyes just failing me or is there no date on this post?
Otherwise, I love this. I was pretty involved in Python testing circles (there was an old Python in tears mailing lists,) and would love to find similar for agentic testing.
kooi | 12 hours ago
Makes me think that LLM assisted formal requirements generation for a roll your own "Aristotle" is possible.
Just thinking about our own projects in the signal processing domain. A nice path could be: human language -> temporal logic specs for black and white box run-time testing.
There are code generators which will turn formal logic into a run time monitor. In the past, I've never used them past toy examples because its so verbose.
> The point of this post is to demonstrate that it’s possible in the era of AI coding to still make decisions based on evidence rather than vibes.
That's the difference between an engineer and a script kiddie.
Domain expertise will always be valuable.
BOOSTERHIDROGEN | 7 hours ago
homarp | an hour ago
59nadir | 7 hours ago
A good motivation to stray from the conventional (non-)wisdom of using Lean as a solution as well, particularly because it's in many cases worse than much less rigorous methods for correctness, and worse than doing literally nothing special at all.
homarp | an hour ago
reedf1 | 12 hours ago
BOOSTERHIDROGEN | 7 hours ago
ArtRichards | 12 hours ago
reedf1 | 12 hours ago
polskibus | 11 hours ago
scotty79 | 11 hours ago
https://danluu.com/pl-tokens/
I'd love to see more of that. I'm currently defaulting to Rust in my software building but F# looks super strong there. On par with Python for correctness with Ultra effort.
I wonder how Rust and other languages are doing in Astra 6/Sol 6.1 and on more projects.
scotty79 | 6 hours ago
vessenes | 5 hours ago
His general test methods this year have been to provide a specification document of some sort and then prompt a clean room implementation of the spec, which can be tested against holdout tests.
Things move pretty quickly - this summer, his test was Zstd and results were at like 25-30% of tests passing at best. He now considers Zstd saturated, as is IMAP from RFC; pandoc is one that he's used throughout, and is not yet perfect.
With all that in mind, read away.
__mharrison__ | 4 hours ago
Otherwise, I love this. I was pretty involved in Python testing circles (there was an old Python in tears mailing lists,) and would love to find similar for agentic testing.
guipsp | an hour ago