The theme for this website is broken on dark mode. When something is highlighted (like with an anchor link) then the background and text are both white.
Great writeup, we need more blogs on writing actual end-to-end tests practically. I did go hard on end-to-end tests for a personal project a few years back, where I initially started with standing up the whole stack via docker-compose but sadly it did not scale. Then I replaced the whole thing with Testing Without Mocks, but I was inexperienced in this methodology so I may not have followed it in spirit.
Right now at work we're exploring Fakes extensively. It's very similar to global mocks, but with contract tests to keep mocks honest. I plan to write on this once it's a bit more widespread, but the theory is solid IMO.
My main takeaway is that the testing story may not be perfect, but it's better than pushing code obliviously, or manually testing until you're bored to death.
Fantastic approach. I did something similar at a previous gig. Never cleaning up, being robust to existing server state, and generating fresh data for each specific test case also helped us, as in this case, to massively parallelize our test suite.
Basically, I write tests just like anyone would use the app in production: each test creates its own objects without relying on any existing data, never touches data it did not create, and never cleans up anything. Data just accumulates. This strategy works really well for apps like Réécoute, where nothing is actually public.
This is my love language ;) Also regarding the discussion of fakes: a lot of people complain about singleton fakes, due to the fact that they won't reflect changes in the real behavior of downstream systems (until you, the developer, become aware of them). That, and the concern about run-times (where it seems playwright parallelization helps a lot) are a common push-back that I get when I adopt this testing approach. (I touch on this a bit in the linked blog post).
META: I wrote up a longer comment previously but got stuck on an Anubis page that wouldn't progress (no bar, no nothing). Clicking 'back' took me back to the lobsters main page. Something is busted, it seems.
each test creates its own objects without relying on any existing data, never touches data it did not create, and never cleans up anything. Data just accumulates.
I don't get this from my experience. At my current job the dev team set up the QA server that way and stuck us with it because "it doubles as load-testing" (which is such an old code smell I felt nostalgic when I heard it) and I feel like "without relying on any existing data" is great, but "without running into conflicts with existing data" is the harder part. I also wonder if you're testing the same thing a year later that you were on day one.
Yeah I definitely think end-to-end feature testing "doubling as load-testing" is a smell. If you're not setting out to load-test in particular, you're not going to stress the system in any meaningful way.
The deal with 'data just accumulating' is that this is strictly a thing in your local setup, so pretty much every time you restart your local env, you should be running with a clean slate. Tests not cleaning up after themselves also helps, because you can inspect the wreckage, so to speak, when a test fails. You can examine the entire state of the system, including the DB.
And also yeah, I think 'not conflicting with existing data' is a bit of an extra challenge, requiring a bit more upfront tooling effort so that unique identifiers and "scopes" (chat groups, projects, whatever widget your system deals with) can be easily brought up for each test case.
gavinmorrow | 7 hours ago
The theme for this website is broken on dark mode. When something is highlighted (like with an anchor link) then the background and text are both white.
Edit: also headings.
[OP] motet-a | 6 hours ago
Indeed! I added this dark mode recently, I forgot to test this part of the site… it should be fixed in a few minutes. Thank you!
frontsideair | 2 hours ago
Great writeup, we need more blogs on writing actual end-to-end tests practically. I did go hard on end-to-end tests for a personal project a few years back, where I initially started with standing up the whole stack via docker-compose but sadly it did not scale. Then I replaced the whole thing with Testing Without Mocks, but I was inexperienced in this methodology so I may not have followed it in spirit.
Right now at work we're exploring Fakes extensively. It's very similar to global mocks, but with contract tests to keep mocks honest. I plan to write on this once it's a bit more widespread, but the theory is solid IMO.
My main takeaway is that the testing story may not be perfect, but it's better than pushing code obliviously, or manually testing until you're bored to death.
pzel | 11 hours ago
Fantastic approach. I did something similar at a previous gig. Never cleaning up, being robust to existing server state, and generating fresh data for each specific test case also helped us, as in this case, to massively parallelize our test suite.
This is my love language ;) Also regarding the discussion of fakes: a lot of people complain about singleton fakes, due to the fact that they won't reflect changes in the real behavior of downstream systems (until you, the developer, become aware of them). That, and the concern about run-times (where it seems playwright parallelization helps a lot) are a common push-back that I get when I adopt this testing approach. (I touch on this a bit in the linked blog post).
META: I wrote up a longer comment previously but got stuck on an Anubis page that wouldn't progress (no bar, no nothing). Clicking 'back' took me back to the lobsters main page. Something is busted, it seems.
trenchant | 8 hours ago
Same thing happened to me yesterday except clicking Reply would only embed a busted Anubis page in place of where the typing box usually appears.
tclancy | 2 hours ago
I don't get this from my experience. At my current job the dev team set up the QA server that way and stuck us with it because "it doubles as load-testing" (which is such an old code smell I felt nostalgic when I heard it) and I feel like "without relying on any existing data" is great, but "without running into conflicts with existing data" is the harder part. I also wonder if you're testing the same thing a year later that you were on day one.
pzel | 2 hours ago
Yeah I definitely think end-to-end feature testing "doubling as load-testing" is a smell. If you're not setting out to load-test in particular, you're not going to stress the system in any meaningful way.
The deal with 'data just accumulating' is that this is strictly a thing in your local setup, so pretty much every time you restart your local env, you should be running with a clean slate. Tests not cleaning up after themselves also helps, because you can inspect the wreckage, so to speak, when a test fails. You can examine the entire state of the system, including the DB.
And also yeah, I think 'not conflicting with existing data' is a bit of an extra challenge, requiring a bit more upfront tooling effort so that unique identifiers and "scopes" (chat groups, projects, whatever widget your system deals with) can be easily brought up for each test case.
icholy | 26 minutes ago
What I've landed on is a small set of critical e2e tests, then MSW http level mocks for everything else.