that seems to be the unconfirmed implication. if true, this will be a hard lesson for anyone trying to do high-visibility research with frontier models and will hopefully drive traffic towards open ones as they catch up. we won’t see a complete brain drain (indirect if mathematicians choose other models) while other strong mathematicians continue to work for the frontier labs, but maybe the complexity proof experts for example will continue to try and prove P not NP without the help of openai or anthropic.
I think the accusation is unfounded, and it's more likely the OpenAI team got wind of the work via the math grapevine. However, it's a measure of OpenAI's reputation that a lot of people will immediately assume they snooped user inputs.
I think it's more a case of a team of mathematicians, probably quite young, who were hired by OpenAI and felt pressure or pressured themselves to deliver results for the mother company.
Do you have a source showing that the prompts you send through ChatGPT are treated as "private" and inaccessible? They seem to have that policy specifically for enterprise agreements, which it doesn't sound like the author worked under. I haven't found anything else.
Strictly speaking it’s unfounded, but the apparent reluctance to share details about the timing of the initiation of the prompts among other details is mildly suspect. I don’t think it’s unreasonable to assume OpenAI keeps tabs on chat traces. It’s trivial to set up AI loops that watch for and build upon existing inquiries given the AI-native introspection infrastructure they have.
I don't think that's what Tristan says based on OpenAI's answers. He also asked about training using his chats and OpenAI didn't answer. I thought it was official that they continuously train on user data and it seems obvious that this can turn anybody's experiments into an output from the model afterwards.
This may be the academic equivalent of what's happening with software security right now.
Just knowing that a piece of software has an undisclosed security bug is enough to point a coding agent at it and find the bug (and maybe a few others, too).
Is the same now true for mathematics? Just knowing that "an LLM solved problem X, result soon to be published" indicates that problem X can be solved, which is enough to point your own reasoning LLMs at it (and they're all "reasoning" these days) to see if you can find the same result.
If you're an AI lab you have effectively unlimited research tokens to spend on those efforts.
Academic secrecy is unhealthy enough already, now we are incentivizing researchers to not even hint at what they're working on less someone else beat them to it.
As I understand it, though, it was a bit more than that. Tristan alludes to them specifically selecting an unusual angle of attack which happened to be the same one he was using. It's more like, there's a kernel bug Somewhere, you have been talking to GPT for weeks about attack strategies for the eBPF interpreter, and as soon as OpenAI knows you in particular found the bug, they immediately hyperfocus on the eBPF interpreter in their internal research to find the bug even though you've never said out loud that that's where the bug is.
alper | a day ago
Do I get it right that OpenAI trawls the prompts of people using the platform for stuff they can use?
anex9d | 22 hours ago
that seems to be the unconfirmed implication. if true, this will be a hard lesson for anyone trying to do high-visibility research with frontier models and will hopefully drive traffic towards open ones as they catch up. we won’t see a complete brain drain (indirect if mathematicians choose other models) while other strong mathematicians continue to work for the frontier labs, but maybe the complexity proof experts for example will continue to try and prove P not NP without the help of openai or anthropic.
gerikson | 21 hours ago
I think the accusation is unfounded, and it's more likely the OpenAI team got wind of the work via the math grapevine. However, it's a measure of OpenAI's reputation that a lot of people will immediately assume they snooped user inputs.
I think it's more a case of a team of mathematicians, probably quite young, who were hired by OpenAI and felt pressure or pressured themselves to deliver results for the mother company.
alper | 21 hours ago
Unless they have significant internal guard rails, there's no telling what data they could/would access under such pressure.
simonw | 19 hours ago
OpenAI are large enough now that I would expect they have extremely stringent access policies and logs for internal access to private date.
bmo | 19 hours ago
Do you have a source showing that the prompts you send through ChatGPT are treated as "private" and inaccessible? They seem to have that policy specifically for enterprise agreements, which it doesn't sound like the author worked under. I haven't found anything else.
anex9d | 19 hours ago
Strictly speaking it’s unfounded, but the apparent reluctance to share details about the timing of the initiation of the prompts among other details is mildly suspect. I don’t think it’s unreasonable to assume OpenAI keeps tabs on chat traces. It’s trivial to set up AI loops that watch for and build upon existing inquiries given the AI-native introspection infrastructure they have.
adrien | 21 hours ago
I don't think that's what Tristan says based on OpenAI's answers. He also asked about training using his chats and OpenAI didn't answer. I thought it was official that they continuously train on user data and it seems obvious that this can turn anybody's experiments into an output from the model afterwards.
simonw | 21 hours ago
Anthropic's analytics system for understanding how people use their service without snooping on their prompts is super interesting. OpenAI described their own, similar system in a 63 page paper which I haven't properly read yet.
simonw | 21 hours ago
This may be the academic equivalent of what's happening with software security right now.
Just knowing that a piece of software has an undisclosed security bug is enough to point a coding agent at it and find the bug (and maybe a few others, too).
Is the same now true for mathematics? Just knowing that "an LLM solved problem X, result soon to be published" indicates that problem X can be solved, which is enough to point your own reasoning LLMs at it (and they're all "reasoning" these days) to see if you can find the same result.
If you're an AI lab you have effectively unlimited research tokens to spend on those efforts.
Academic secrecy is unhealthy enough already, now we are incentivizing researchers to not even hint at what they're working on less someone else beat them to it.
[OP] pie_flavor | 21 hours ago
As I understand it, though, it was a bit more than that. Tristan alludes to them specifically selecting an unusual angle of attack which happened to be the same one he was using. It's more like, there's a kernel bug Somewhere, you have been talking to GPT for weeks about attack strategies for the eBPF interpreter, and as soon as OpenAI knows you in particular found the bug, they immediately hyperfocus on the eBPF interpreter in their internal research to find the bug even though you've never said out loud that that's where the bug is.
gerikson | a day ago
Mastodon link already posted: https://lobste.rs/s/ki3ylq/finite_time_blowup_with_smooth_forcing