I really dislike this as well, and I’m not a supporter of LLMs, but some humans can also do this without any LLM. Maybe you can simply say that it’s over-engineered and that the tests are bad. I think it’s a valid reason to refuse to merge a PR.
But to answer your question, I have used LLMs to write both tests and code a few times…
Yes. Tests are just code which should be heavily constrained to be:
simple
short
linear
WET not DRY
deterministic
explanatory rather than algorithmic
non tautological
descriptive of the failure mode (i.e. don't assert(foo.is_empty) instead assert_eq(foo, []))
covering the useful branches of the implementation
usually correct (exception: characterization tests used to note drift prior to fixing a bug)
The tests that come out of the default models sometimes tend to not quite fit this, but most frontier models respond pretty well to steering to get this right, so invest in a good set of rules that match your domain on this. There's lots of simple rules you can build up to write better tests in any language.
The thing about tests and LLMs is that you can use just two of the things spec + impl or spec + test in isolation and have the model reason over them for whether they're correct. This is a sort of triangulation type thing that you do as a developer and it's reasonable to do with an LLM too and not just treat it like a black box that summons code into being in one incantation.
The other thing you can do is have the LLM quickly go through the process of testing the Red/Green steps in TDD (whether pre-testing or post-testing) to ensure that a failure shows up somehow and is correctly indicating what went wrong.
LLMs are better at this than humans, assuming they get decent inputs upfront.
I went from your stance to another one: you can generate both with LLMs, but you have to prove that the tests are properly testing your code. This can be done by surgically breaking the implementation for each assertion and check if the test properly fails.
Since it takes a lot of time to do by hand, I'm working on a skill to ease the process.
Another thing that could help is mutation testing.
I do have LLMs generate both implementation and tests. But I spend a lot of time thinking about what needs to be tested and how, to give me confidence the implementation is correct. I.e. I don't give the LLM free reign in deciding what and how to test.
I'm actively experimenting with methods for getting verification right at a pace faster than I can personally guarantee all of the implementation. Things like STAMP/STPA, mutation testing, fuzzing etc. that aim to make verification sliyghtly more algorithmic could be fruitful ways forward. But it's still early days and I have to step in and manually correct a lot.
Steering the model to generate professional grade code and not over generating for the sake of generating.
In all my task, I describe a goal which is a test that obviously it writes. Even with natural language, its code after all.
I tend to refactor myself because writing the prompt that a llm would understand take for me now more times that using the ide facilities. Need to work on that.
I'm still in the "haven't used LLM's to generate any code" camp. The most I've used a LLM for was to generate a backstory for a D&D character. I avoid that plague as much as I can.
To me a lot of LLM tests feel like "scaffolding" - they are not written like you would write a test you intend to keep, but may have helped it validate something that does not need to continually be validated once the code is done.
As such, I have arranged things such that most of the tests it writes wind up in a directory that is, by default, gitignored, and it requires more justification to put them where they'll be checked in.
For throw-away code (HTML dashboards, code that generates graphs for slides, etc.) I don't bother looking at the code at all. The LLM writes tests and I presume it's useful to avoid regressions, but again - I don't really look.
It’s situational. For testing math stuff or CRUD or just making a regression test that’s representative of the bug being fixed, LLM making the code and the test has seemed to work OK especially after doing the LLM meta optimization dance (mutation testing, adversarial review). And we still make decisions around preventing code bloat like if the mocks get excessive not accepting those tests or at least having a conversation about it.
For any tests doing something really useful we’ve made those system tests or dockerized integration tests in a multi service environment pointed at a simulator and due to the business context those have usually been driven by humans either prompting for specific cases or handing off a test plan.
I could see even more delegation becoming possible (having the AI drive integration test case generation) as we are feeding even more context to the LLM as in emails, docs etc. Already we’re finding the LLM output improving a lot when we give this extra context. That flows down to even more useful tests getting generated.
The other day I reviewed a PR for one of my projects that was vibe-coded. The PR comprised a single commit with a small fix and lots of additional tests. The single commit means there's no (easy) way to really make sure all those new tests would fail without the fix.
I think it would have been better to have the agent first write failing tests, commit that, then work on the fix and commit it separately. While I don't vibe-code myself, I think following a strict TDD methodology can make a big difference in this respect.
motet-a | 16 hours ago
I really dislike this as well, and I’m not a supporter of LLMs, but some humans can also do this without any LLM. Maybe you can simply say that it’s over-engineered and that the tests are bad. I think it’s a valid reason to refuse to merge a PR.
But to answer your question, I have used LLMs to write both tests and code a few times…
joshka | 11 hours ago
Yes. Tests are just code which should be heavily constrained to be:
assert(foo.is_empty)insteadassert_eq(foo, []))The tests that come out of the default models sometimes tend to not quite fit this, but most frontier models respond pretty well to steering to get this right, so invest in a good set of rules that match your domain on this. There's lots of simple rules you can build up to write better tests in any language.
The thing about tests and LLMs is that you can use just two of the things spec + impl or spec + test in isolation and have the model reason over them for whether they're correct. This is a sort of triangulation type thing that you do as a developer and it's reasonable to do with an LLM too and not just treat it like a black box that summons code into being in one incantation.
The other thing you can do is have the LLM quickly go through the process of testing the Red/Green steps in TDD (whether pre-testing or post-testing) to ensure that a failure shows up somehow and is correctly indicating what went wrong.
LLMs are better at this than humans, assuming they get decent inputs upfront.
nesk | 11 hours ago
I went from your stance to another one: you can generate both with LLMs, but you have to prove that the tests are properly testing your code. This can be done by surgically breaking the implementation for each assertion and check if the test properly fails.
Since it takes a lot of time to do by hand, I'm working on a skill to ease the process.
Another thing that could help is mutation testing.
LolPython | 8 hours ago
No harm in making a skill but I have had some success getting LLMs to do that by asking it to do manual mutation testing
kqr | 15 hours ago
I do have LLMs generate both implementation and tests. But I spend a lot of time thinking about what needs to be tested and how, to give me confidence the implementation is correct. I.e. I don't give the LLM free reign in deciding what and how to test.
I'm actively experimenting with methods for getting verification right at a pace faster than I can personally guarantee all of the implementation. Things like STAMP/STPA, mutation testing, fuzzing etc. that aim to make verification sliyghtly more algorithmic could be fruitful ways forward. But it's still early days and I have to step in and manually correct a lot.
tomleo | 19 hours ago
Sometimes code & unit tests, then I write the integration & e2e tests. It depends on the code & domain.
gerardnico | 9 hours ago
That’s the core of software engineering, now.
Steering the model to generate professional grade code and not over generating for the sake of generating.
In all my task, I describe a goal which is a test that obviously it writes. Even with natural language, its code after all.
I tend to refactor myself because writing the prompt that a llm would understand take for me now more times that using the ide facilities. Need to work on that.
miah | 6 hours ago
I'm still in the "haven't used LLM's to generate any code" camp. The most I've used a LLM for was to generate a backstory for a D&D character. I avoid that plague as much as I can.
cr8 | 4 hours ago
To me a lot of LLM tests feel like "scaffolding" - they are not written like you would write a test you intend to keep, but may have helped it validate something that does not need to continually be validated once the code is done.
As such, I have arranged things such that most of the tests it writes wind up in a directory that is, by default, gitignored, and it requires more justification to put them where they'll be checked in.
freddyb | 3 hours ago
For throw-away code (HTML dashboards, code that generates graphs for slides, etc.) I don't bother looking at the code at all. The LLM writes tests and I presume it's useful to avoid regressions, but again - I don't really look.
LolPython | 12 hours ago
It’s situational. For testing math stuff or CRUD or just making a regression test that’s representative of the bug being fixed, LLM making the code and the test has seemed to work OK especially after doing the LLM meta optimization dance (mutation testing, adversarial review). And we still make decisions around preventing code bloat like if the mocks get excessive not accepting those tests or at least having a conversation about it.
For any tests doing something really useful we’ve made those system tests or dockerized integration tests in a multi service environment pointed at a simulator and due to the business context those have usually been driven by humans either prompting for specific cases or handing off a test plan.
I could see even more delegation becoming possible (having the AI drive integration test case generation) as we are feeding even more context to the LLM as in emails, docs etc. Already we’re finding the LLM output improving a lot when we give this extra context. That flows down to even more useful tests getting generated.
Scary times!
kornel | 4 hours ago
You can't do both in a single session, because LLMs will write weaksauce tests to make its code look good.
It works better if you start a new session and give it a goal to find bugs in the code and prove that via tests.
noteflakes | 3 hours ago
The other day I reviewed a PR for one of my projects that was vibe-coded. The PR comprised a single commit with a small fix and lots of additional tests. The single commit means there's no (easy) way to really make sure all those new tests would fail without the fix.
I think it would have been better to have the agent first write failing tests, commit that, then work on the fix and commit it separately. While I don't vibe-code myself, I think following a strict TDD methodology can make a big difference in this respect.