This looks very similar to the claim that distilling a model from Anthropic is the same thing as Anthropic distilling the model from information on the internet.
Which is very flawed, since distillation requires the thing to exist in the thing it’s distilled from. And no LLM model existed in the information Anthropic used to train the model. Instead the model was built using information and utilizing new technology including hardware, software, transformer architecture, etc.
I don't believe I understand your argument? Are you claiming a moral, legal or practical difference? Or are you saying that Anthropic spending resources on training an LLM is somehow different from an author spending resources on writing a book?
In any case, I doubt Kimi was trained without "stealing" the same data. Assembling all of your training data from Claude responses seems infeasible. It's much more likely that Kimi's base model was trained similarly to any other base model, with terabytes of data from all imaginable sources. Then the model was fine-tuned with "high-quality" data, followed by reinforcement learning. Throwing in lots of chat transcripts from other chatbots into the "high-quality" dataset would be expected, and is done to some degree by everyone, but maybe a lot more for Kimi. And likely they did a lot of reinforcement learning against the Claude API
The model would exist without Claude, it just wouldn't be nearly as coherent or smart
All ML/AI models are comparable to some form of compression(al beit lossy) of information and in this case copyrighted information. The OP is pointing to this as stolen data(by all the companies that started with pre trained models)
Maybe both are ok. Maybe neither. Maybe one. The point is this does not follow
>If training on copyrighted data without authors consent is ok then distilling is ok as well.
Because building a model from distillation and building a model from raw data are not the same. You have to evaluate them independently. And legally its different as well. IP vs ToS (civil).
Think of it in terms of distilled knowledge, not distilled LLMs.
I find both claims unsound, though. Knowledge or model behavior itself is not copyrightable, so all these claims just boil down to the "I am not happy with that" argument. You cannot claim someone is stealing something you don't own in the first place.
You misunderstand the point. You can think both or either are morally right/wrong or good/bad for society. But distilling a model and building one using information online are fundamentally different. Even of you think the information the frontier models used was not fairly accessed.
I don't see any US labs suing their Chinese counterparts. It is practically impossible. That makes it a verbal battle, not a legal one.
From a strictly moral standpoint, it is illogical to state, "You stole things from my archive of stolen goods." LLM vendors need to morally own the knowledge before their accusations hold. It is unsound to claim ownership of something resulting from stolen property, regardless of the work you put into building it.
Either those accusations don't hold at all, all training is just "fair use", or the two processes you described are just different forms of "stealing."
Bottom line, even if we consider knowledge copyrightable, it is just stolen goods changing hands. You cannot claim ownership of a derivative work if you deny ownership to the sources your work was derived from. No one holds a higher moral standing than another.
You might as well say a photocopy of a book was "built using information and utilizing new technology including hardware, software, transformer architecture, etc."
I didn’t participate in the discussion yesterday because I find it implausible Fable was available long enough (3-4 weeks cumulative?) to get data and train and have it fundamentally affect.
But I don’t grok training enough to know that’s silly.
If my new prior is you can…that’s a pretty thin moat that’s essentially indefensible.
My guess is they used the other Anthropic models extensively for synthetic data generation. The top most similar models for K3 are (lower means more similar)
Fable 5 -> 0.42
Opus 4.8 -> 0.45
Sonnet 5 -> 0.45
Opus 4.7 -> 0.46
Grok 4.3 -> 0.52
There's an obvious jump at Grok 4.3, and it would not surprise me that the similarity there is because Grok used Anthropic models for training too (you can get the similarity list for Grok and it does look like the top most similar models for Grok are either Anthropic models or some Chinese models).
The damning evidence that K3 used Anthropic models for training is that K3 is more similar to those models than it is to K2.6. If you look at the Anthropic, OpenAI or Google models, they are most similar with their own other models. Not so with K3, where K2.6 is less similar than 15 other models.
Now, why is Fable 5 the most similar to K3 and not Opus 4.8. I think it's quite likely that K3 did some fine tuning at the end, when Fable 5 became available. They probably had all the infrastructure in place, and Mythos had been announced for months, so they were probably waiting for the second the newest Anthropic model was released to start using it for synthetic data generation.
Note that until last generation all Chinese models were relatively small. This was mainly due to lack of training hardware, but as soon as the new Huawei NPUs started shipping some Chinese labs switched to larger models:
Deepseek: 670B to 1.6T
Moonshot: 1.1T to 2.8T
Xiaomi: 310B to 1T
The new models also use a different architecture so I would assume in tests they will look different from previous generations.
Z.ai (GLM) and MiniMax on the other hand have continued training existing models (with some surgical changes to improve long context memory) so they should score similarly in these tests.
All frontier models have been trained without any regards for IP protection laws. I don't see how anyone can argue in good faith that distillation is not fair game and does not ultimately "benefit humanity™"
I get what you're saying and two wrongs do not make a right but the irony, and why people are even talking about this, is that the thing being distilled clearly, and knowingly, violated copyright & terms across the entire internet.
Training a model is very expensive and creates something no individual rights-holder could. Distilling a model copies this value add and captures it without bearing the cost that created it.
They only had to pay for storing the books on a server for later possible use. They did not have to pay anything for the training which was declared fair use.
This actually undermines the argument that distilling is harmless because its founded on the idea that Anthropic did the same thing and didn’t have any repercussions.
the outputs of the model have no property protections, and training a model on the outputs of another model does the exact same thing - its expensive and creates new value over what was in the input - a set of documents.
Could you give an example of the value that only training a model can create but none of the rights-holder could? I feel like if you got a direct, instant communication channel to any of the rights-holder that created the content in the training set of those models, you'd get more value than what the LLM could ever give you on any specific subject.
I understand you think Anthropic should have paid for the information it trained the models on. But im talking about all the costs to build a model. Do you think Anthropic didnt spend money to build these models? Did you not know that its actually very expensive?
For me, I don't really care about the theft aspect but when people are claiming that these open models are better value or going to overtake anthropic/openai models, the implication that the open models are training of distilled data means all the "progress" they are making is just mimiced from the closed models.
It's a bit interesting how the open models are able to keep pace with the closed models except whole maintaining a steady following time.
Distillation is absolutely not the reason they're good. It's not necessarily even done on a more capable model. It can even be done on itself and still bring improvement, or on a weaker model as well (see GLM and Gemini, which is definitely true because it repeats Deepmind's injections).
It's important to note that even if these open models are distilled, they are showing genuine improvements in their architecture, which enables inference costs to be several factors below what equivalent closed models have.
The interesting question is: will Anthropic release a Fable like model with an architecture similar to Kimi, and get the inference cost gains? They should surely beat Kimi because they can internally distill as much as they want.
This data sort of disqualifies itself: unless Moonshot has a time machine, K3 should be more similar to Opus 4.5-4.8 than Fable 5.
Keep in mind, Anthropic started limiting access and introduced anti-distillation measures around 4.5-4.6 (?). So the majority of distillation should have happened on earlier models.
Maybe a better explanation is that they have access to the same training datasets? Which if private can again raise questions about theft, but on a very different level.
I also suspect the questions asked matter a lot, and the system prompts matter a lot, because "the map is built from nothing but the words they choose" - so this is more a measure of linguistic style than anything.
If you use the Claude Code harness on two models you will probably get very similarly styled output. I would not be surprised if K3 "stole" a lot of the harness that was leaked.
Notice how the only rows that even come close to being gray is GPT-5.4+Mini, diverging even from other GPT models. Is this because it has a wholly different training set, or (more likely IMO), did it just have a system prompt that leads to a different style?
given that github is full of vibe coded repos, reddit is full of bot comments, and most blog posts are ai slop, isnt it guaranteed that anyone training on big public data sets will be closely tracking each other in text style?
If I am working on something not sensitive and afterwards upload my chat log to a public dataset, can open models use that data for training?
Keep in mind, I paid for access to model and half of the work is mine (the prompts). But can the model provider forbid me from publishing my result as open data?
Worthless. Any LLM output similarity metric that uses n-grams as a source might as well measure the average temperature on Mars surface, no matter how much lipstick you put on it. It just can't have enough certainty. There used to be an n-gram benchmark popular on Twitter that showed extreme similarity of grok-3-beta to gpt-4.5-preview, while these models were trained on new base ones, came out 2 weeks apart, and were unmistakably different. Results were wildly inconsistent run to run. It didn't stop the crowd believing its creator in that DeepSeek R1 was trained on o1-preview (which was obvious bullshit as well, they were as different as two models can be). It's amazing how you can put anything on the web and everybody will believe you without checking or even understanding of what they're looking at.
K3 was trained on Claude's outputs, though - it repeats Anthropic's prompt injections 1:1 in its reasoning, which you should know if you ever tinkered with both models long enough. Good for them.
great_psy | 5 hours ago
There is an optimal answer to any question. Something that maximizes utility and minimizes tokens.
isoprophlex | 5 hours ago
nonethewiser | 5 hours ago
This looks very similar to the claim that distilling a model from Anthropic is the same thing as Anthropic distilling the model from information on the internet.
Which is very flawed, since distillation requires the thing to exist in the thing it’s distilled from. And no LLM model existed in the information Anthropic used to train the model. Instead the model was built using information and utilizing new technology including hardware, software, transformer architecture, etc.
wongarsu | 5 hours ago
In any case, I doubt Kimi was trained without "stealing" the same data. Assembling all of your training data from Claude responses seems infeasible. It's much more likely that Kimi's base model was trained similarly to any other base model, with terabytes of data from all imaginable sources. Then the model was fine-tuned with "high-quality" data, followed by reinforcement learning. Throwing in lots of chat transcripts from other chatbots into the "high-quality" dataset would be expected, and is done to some degree by everyone, but maybe a lot more for Kimi. And likely they did a lot of reinforcement learning against the Claude API
The model would exist without Claude, it just wouldn't be nearly as coherent or smart
ghm2199 | 5 hours ago
nonethewiser | 2 hours ago
yreg | 5 hours ago
If training on copyrighted data without authors consent is ok then distilling is ok as well.
nonethewiser | 2 hours ago
>If training on copyrighted data without authors consent is ok then distilling is ok as well.
Because building a model from distillation and building a model from raw data are not the same. You have to evaluate them independently. And legally its different as well. IP vs ToS (civil).
[OP] maxloh | 5 hours ago
I find both claims unsound, though. Knowledge or model behavior itself is not copyrightable, so all these claims just boil down to the "I am not happy with that" argument. You cannot claim someone is stealing something you don't own in the first place.
nonethewiser | 2 hours ago
[OP] maxloh | 39 minutes ago
From a strictly moral standpoint, it is illogical to state, "You stole things from my archive of stolen goods." LLM vendors need to morally own the knowledge before their accusations hold. It is unsound to claim ownership of something resulting from stolen property, regardless of the work you put into building it.
Either those accusations don't hold at all, all training is just "fair use", or the two processes you described are just different forms of "stealing."
Bottom line, even if we consider knowledge copyrightable, it is just stolen goods changing hands. You cannot claim ownership of a derivative work if you deny ownership to the sources your work was derived from. No one holds a higher moral standing than another.
jayd16 | 5 hours ago
nonethewiser | 2 hours ago
codelion | 5 hours ago
shiandow | 5 hours ago
Edit: No that doesn't seem to be what's happening here. I think it's some kind of word frequency analysis.
josh-paul | 5 hours ago
weego | 5 hours ago
zobzu | 5 hours ago
refulgentis | 5 hours ago
But I don’t grok training enough to know that’s silly.
If my new prior is you can…that’s a pretty thin moat that’s essentially indefensible.
credit_guy | 4 hours ago
Fable 5 -> 0.42
Opus 4.8 -> 0.45
Sonnet 5 -> 0.45
Opus 4.7 -> 0.46
Grok 4.3 -> 0.52
There's an obvious jump at Grok 4.3, and it would not surprise me that the similarity there is because Grok used Anthropic models for training too (you can get the similarity list for Grok and it does look like the top most similar models for Grok are either Anthropic models or some Chinese models).
The damning evidence that K3 used Anthropic models for training is that K3 is more similar to those models than it is to K2.6. If you look at the Anthropic, OpenAI or Google models, they are most similar with their own other models. Not so with K3, where K2.6 is less similar than 15 other models.
Now, why is Fable 5 the most similar to K3 and not Opus 4.8. I think it's quite likely that K3 did some fine tuning at the end, when Fable 5 became available. They probably had all the infrastructure in place, and Mythos had been announced for months, so they were probably waiting for the second the newest Anthropic model was released to start using it for synthetic data generation.
throwa356262 | 4 hours ago
Deepseek: 670B to 1.6T
Moonshot: 1.1T to 2.8T
Xiaomi: 310B to 1T
The new models also use a different architecture so I would assume in tests they will look different from previous generations.
Z.ai (GLM) and MiniMax on the other hand have continued training existing models (with some surgical changes to improve long context memory) so they should score similarly in these tests.
pandoro | 5 hours ago
skybrian | 5 hours ago
tills13 | 4 hours ago
skybrian | 3 hours ago
Laurel1234 | 3 hours ago
nonethewiser | 5 hours ago
mosura | 5 hours ago
yandie | 5 hours ago
I don’t feel sorry for the model companies
mosura | 5 hours ago
nonethewiser | 2 hours ago
applicative | 4 hours ago
mosura | 4 hours ago
nonethewiser | 2 hours ago
titzer | 5 hours ago
jayd16 | 5 hours ago
8note | 4 hours ago
pandoro | 4 hours ago
nonethewiser | 2 hours ago
Yeah.
LLMs
shlewis | 4 hours ago
nonethewiser | 2 hours ago
rvba | 4 hours ago
verdverm | 3 hours ago
applicative | 4 hours ago
parineum | 4 hours ago
It's a bit interesting how the open models are able to keep pace with the closed models except whole maintaining a steady following time.
orbital-decay | 3 hours ago
abdullahkhalids | 3 hours ago
The interesting question is: will Anthropic release a Fable like model with an architecture similar to Kimi, and get the inference cost gains? They should surely beat Kimi because they can internally distill as much as they want.
throwa356262 | 5 hours ago
Keep in mind, Anthropic started limiting access and introduced anti-distillation measures around 4.5-4.6 (?). So the majority of distillation should have happened on earlier models.
Maybe a better explanation is that they have access to the same training datasets? Which if private can again raise questions about theft, but on a very different level.
sebastianconcpt | 4 hours ago
josefritzishere | 4 hours ago
causal | 4 hours ago
K3-to-Fable is blue at 0.42. Is 0.42 meaningful, or did we set 0.4 as the lower bound because it makes 0.42 look significant?
Sol-to-Fable is 0.69. It's dark yellow, making this look VERY different from 0.42. But is it? What do these numbers mean in absolute terms?
causal | 4 hours ago
If you use the Claude Code harness on two models you will probably get very similarly styled output. I would not be surprised if K3 "stole" a lot of the harness that was leaked.
Notice how the only rows that even come close to being gray is GPT-5.4+Mini, diverging even from other GPT models. Is this because it has a wholly different training set, or (more likely IMO), did it just have a system prompt that leads to a different style?
c0n5pir4cy | 4 hours ago
8note | 4 hours ago
Fokamul | 4 hours ago
throwa356262 | 4 hours ago
If I am working on something not sensitive and afterwards upload my chat log to a public dataset, can open models use that data for training?
Keep in mind, I paid for access to model and half of the work is mine (the prompts). But can the model provider forbid me from publishing my result as open data?
kingstnap | 4 hours ago
So who's training on who's outputs?
https://news.ycombinator.com/item?id=48990086
https://x.com/stevibe/status/2026227392076018101
orbital-decay | 4 hours ago
Worthless. Any LLM output similarity metric that uses n-grams as a source might as well measure the average temperature on Mars surface, no matter how much lipstick you put on it. It just can't have enough certainty. There used to be an n-gram benchmark popular on Twitter that showed extreme similarity of grok-3-beta to gpt-4.5-preview, while these models were trained on new base ones, came out 2 weeks apart, and were unmistakably different. Results were wildly inconsistent run to run. It didn't stop the crowd believing its creator in that DeepSeek R1 was trained on o1-preview (which was obvious bullshit as well, they were as different as two models can be). It's amazing how you can put anything on the web and everybody will believe you without checking or even understanding of what they're looking at.
K3 was trained on Claude's outputs, though - it repeats Anthropic's prompt injections 1:1 in its reasoning, which you should know if you ever tinkered with both models long enough. Good for them.