I think a lot of people (myself included) have objections to AI that are unrelated, or in addition, to its (lack of) capabilities. Even if an LLM might find bugs, they might feel that this is not worth the widespread personal and societal cost of using and training them. Yes, that means that a library might have bugs that an LLM may have helped remove, but individual engineers will have to offset that against the impacts of LLM use and make their own judgment.
If that's the only thing holding you back, it's not a big issue - there's regularly a new model being tested in stealth for free. With a harness configured for review all of them should be good enough these days. For example see https://openrouter.ai/models?variant=free&output_modalities=text or https://open-code.ai/en/docs/zen#pricing For open source projects you can also use the free tiers used for data collection (the code was already ingested anyway)
Right, but they don't parade with these free offerings, you have to know they exist. Plus you have to figure out if you trust this company with your PII and credit card and all the input/output it processes. We're talking first time users that probably never had an interaction with any LLM providers.
Also: lots of people go entire lives without a credit card! I feel like credit cards are an incredibly US-centric thing.
Also: I found harnesses took real effort to learn initially. Sure you can run something and type a broad question as prompt, but it's more about: figuring out if you can trust this piece of software and this LLM thing with broad access to your system. (Or wrangling a VM into submission for this possible one-off.)
Arguably, Codex's free tier is the easiest path, because I believe you can see it in action without an account and get good results.
For professional work at a company that already uses AI? Sure, let the machine loose. I think it's one of the better uses of LLMs to only use them as pre-human review.
For a hobby FOSS project, where the author is not interested in using AI? THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS”.
Yes, it is entirely too much to ask. Entitled consumers of FOSS projects are nothing new, if you're not paying for something, you have no right to make demands.
And let's get real, most reasonably long-lived projects contain technical debt galore. You can point an AI or linter or whatever at it and come up with a long list of stuff to fix. But since it works well enough, such stuff is low-priority. Even in settings where people get paid to do it, it usually never gets fixed. In a FOSS project, you can't really ask someone to spend their free time to do it, so either do it yourself or live with it.
Most open source projects are labours of love. They exist because the maintainers like doing it.
Once it starts feeling like work, chances are high that the project gets abandoned or that the maintainer takes a longer break from it.
AI can be helpful, but it can also uncover false positives. It can discover race conditions in code that will never experience that race condition in practice, or simply misunderstand how something works, or outright make something up.
Verifying and fixing something that an AI spits out doesn’t seem particularly interesting to spend ones unpaid, personal time on. And it’s worth pointing out that people who don’t want AI involvement in the first place, probably aren’t interested in paying for it either.
But I want to use your library, and when I ask my AI to evaluate it for bugs, memory leaks, security issues, it returns a long list of serious ones.
Not on my library it does not. I did lately get bug reports that looked AI generated, all bogus, all based on misuse of the API, and some suggesting fixes that were even worse than the current situation. Which I reckon is not ideal, I'd rather have an impossible to misuse API, or at least total functions. But this is C we're talking about, there's little choice there.
However, this is worth your attention given the nature of the project:
Most important finding: the download on the homepage is the vulnerable version. The homepage still links monocypher-4.0.2.tar.gz, but the changelog and bugs page say 4.0.2 and below have a timing leak in EdDSA/Ed25519 signing (compiler-dependent)
I just did a silly "check this". I have no idea how to properly ask the AI to evaluate a project I'm not involved in (esp on a topic my capacity is limited). And I'm not familiar with the code, so I have no idea how to evaluate what Fable reported: https://claude.ai/share/b056adee-c43b-4b76-9017-2a13d7311b76
Maybe all of the findings are trash, maybe one of them is of some value to you. But that's my point, it doesn't hurt using an AI to check the code, does it?
The deeper audit of <redacted> found a core bug reachable through valid Unicode text, despite upstream tests passing in Debug and ReleaseFast. Adoption unchanged is discouraged; the next step is to investigate the bug and assess fixes.
That's a u8 holding values that can potentially be 4 or even 8 bytes.
Am I supposed to let the AI fix it and submit a PR? Will I be accused of AI-slop?
I have an open source project that gets contributions every once in a while, here’s my perspective.
Opening a PR is only acceptable if:
The project accepts PRs assisted by AI.
you’ve personally verified that the PR fixes an actual problem, and that the fix is easy to review.
The description is easy to read, and describes the issue well (AI is bad at this, do it by hand).
If 1 is ok, but you skip 2 and 3, you’re just adding work to the maintainers, and most of them will not find that helpful.
Same thing applies to making issues.
Pointing an AI at a repo is a start. The real work is ruling out false positives, coming up with the good solution out of a pool of all solutions, and making the end result understandable to others is where most of the work is.
I am an engineer. And I totally agree "coming up with the good solution out of a pool of all solutions, and making the end result understandable to others is where most of the work is" (that's why I don't mind a computer doing the tedious part).
And I could just vendor the library, fix the bug, and solve my problem.
What bothers me is
a) I need to get into these weird politics, which I don't understand. There is no clear no-AI contribution policy, but it's on codeberg, does this mean something? Do I have to dig into some cultural war just to say, "my friend, this u8 should be u32 or u64 or even better, uoffset, thank you for maintaining this"?
b) Other devs using this library introduce a bug to their project that could have been fixed.
Btw, this is one of the simpler cases, in other cases bugs can be deeper and more complex, but also more severe.
hgrsd | an hour ago
I think a lot of people (myself included) have objections to AI that are unrelated, or in addition, to its (lack of) capabilities. Even if an LLM might find bugs, they might feel that this is not worth the widespread personal and societal cost of using and training them. Yes, that means that a library might have bugs that an LLM may have helped remove, but individual engineers will have to offset that against the impacts of LLM use and make their own judgment.
stephank | an hour ago
You know that stuff costs money, right?
viraptor | an hour ago
If that's the only thing holding you back, it's not a big issue - there's regularly a new model being tested in stealth for free. With a harness configured for review all of them should be good enough these days. For example see https://openrouter.ai/models?variant=free&output_modalities=text or https://open-code.ai/en/docs/zen#pricing For open source projects you can also use the free tiers used for data collection (the code was already ingested anyway)
stephank | 39 minutes ago
Right, but they don't parade with these free offerings, you have to know they exist. Plus you have to figure out if you trust this company with your PII and credit card and all the input/output it processes. We're talking first time users that probably never had an interaction with any LLM providers.
Also: lots of people go entire lives without a credit card! I feel like credit cards are an incredibly US-centric thing.
Also: I found harnesses took real effort to learn initially. Sure you can run something and type a broad question as prompt, but it's more about: figuring out if you can trust this piece of software and this LLM thing with broad access to your system. (Or wrangling a VM into submission for this possible one-off.)
Arguably, Codex's free tier is the easiest path, because I believe you can see it in action without an account and get good results.
[OP] vrypan | an hour ago
What's more, I get it. Can I run it for you and submit the fix? Or is it not acceptable because it's AI-generated?
spc476 | 38 minutes ago
If the PR contains input and/or a test case showing the actual bug, and the code follows the coding convention then maybe I won't outright reject it.
nemin | 2 hours ago
Depends.
For professional work at a company that already uses AI? Sure, let the machine loose. I think it's one of the better uses of LLMs to only use them as pre-human review.
For a hobby FOSS project, where the author is not interested in using AI? THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS”.
kryptiskt | 57 minutes ago
Yes, it is entirely too much to ask. Entitled consumers of FOSS projects are nothing new, if you're not paying for something, you have no right to make demands.
And let's get real, most reasonably long-lived projects contain technical debt galore. You can point an AI or linter or whatever at it and come up with a long list of stuff to fix. But since it works well enough, such stuff is low-priority. Even in settings where people get paid to do it, it usually never gets fixed. In a FOSS project, you can't really ask someone to spend their free time to do it, so either do it yourself or live with it.
[OP] vrypan | 36 minutes ago
Come on my friend. I wrote my first code in the 80s, and I set up my first linux machine from a pack of floppy disks.
robinheghan | an hour ago
Most open source projects are labours of love. They exist because the maintainers like doing it.
Once it starts feeling like work, chances are high that the project gets abandoned or that the maintainer takes a longer break from it.
AI can be helpful, but it can also uncover false positives. It can discover race conditions in code that will never experience that race condition in practice, or simply misunderstand how something works, or outright make something up.
Verifying and fixing something that an AI spits out doesn’t seem particularly interesting to spend ones unpaid, personal time on. And it’s worth pointing out that people who don’t want AI involvement in the first place, probably aren’t interested in paying for it either.
Loup-Vaillant | 28 minutes ago
Not on my library it does not. I did lately get bug reports that looked AI generated, all bogus, all based on misuse of the API, and some suggesting fixes that were even worse than the current situation. Which I reckon is not ideal, I'd rather have an impossible to misuse API, or at least total functions. But this is C we're talking about, there's little choice there.
[OP] vrypan | 11 minutes ago
Heh, yes, that's a really well-built library.
However, this is worth your attention given the nature of the project:
I just did a silly "check this". I have no idea how to properly ask the AI to evaluate a project I'm not involved in (esp on a topic my capacity is limited). And I'm not familiar with the code, so I have no idea how to evaluate what Fable reported: https://claude.ai/share/b056adee-c43b-4b76-9017-2a13d7311b76
Maybe all of the findings are trash, maybe one of them is of some value to you. But that's my point, it doesn't hurt using an AI to check the code, does it?
[OP] vrypan | 2 hours ago
Here's an example:
That's a u8 holding values that can potentially be 4 or even 8 bytes.
Am I supposed to let the AI fix it and submit a PR? Will I be accused of AI-slop?
robinheghan | an hour ago
I have an open source project that gets contributions every once in a while, here’s my perspective.
Opening a PR is only acceptable if:
If 1 is ok, but you skip 2 and 3, you’re just adding work to the maintainers, and most of them will not find that helpful.
Same thing applies to making issues.
Pointing an AI at a repo is a start. The real work is ruling out false positives, coming up with the good solution out of a pool of all solutions, and making the end result understandable to others is where most of the work is.
[OP] vrypan | an hour ago
I am an engineer. And I totally agree "coming up with the good solution out of a pool of all solutions, and making the end result understandable to others is where most of the work is" (that's why I don't mind a computer doing the tedious part).
And I could just vendor the library, fix the bug, and solve my problem.
What bothers me is
a) I need to get into these weird politics, which I don't understand. There is no clear no-AI contribution policy, but it's on codeberg, does this mean something? Do I have to dig into some cultural war just to say, "my friend, this u8 should be u32 or u64 or even better, uoffset, thank you for maintaining this"?
b) Other devs using this library introduce a bug to their project that could have been fixed.
Btw, this is one of the simpler cases, in other cases bugs can be deeper and more complex, but also more severe.