LoRA + SFT, but it'll be big - better to wait for a finetuning API from one of the providers, I wouldn't jump straight to RL or off-policy pseudo-RL like DPO.
Open Source models decelerate growth of closed AI. For people who think (or want) AI = closed_AI then that argument has weight. Good luck getting them to update their priors.
Open source AI is actually a lot less "powerful" than genuine frontier models, i.e. it has a much tighter inherent capability ceiling. This is "decelerationist" from a purely AGI-pilled point of view but it's actually great if you're worried about a capabilities arms race putting AI Safety at severe risk.
Kimi K3 is plausibly a lot less dangerous than a totally jailbroken ChatGPT/Gemini/Claude Sonnet (let alone Opus or Fable!) and it's quite deeply weird how no one seems to be calling for those models to be banned or restrained by further regulation. Why the double standard against the less concerning (but more efficient!) open weight models?
If you're targeting widespread local/on prem deployment which is what many open weight models are doing, that inherently limits your scale in terms of total model weights/inference-time compute compared to running in a few centralized datacenters. A centralized model will always be able to leverage a larger scale of deployment, placing it much closer to the genuine "frontier".
I think the argument is that decentralization leads to deceleration because it means less centralized funding and data. Those are the two primary ingredients for accel.
The problem with the decel/accel rhetoric is that it lacks nuance.
If your worldview is “most of the progress is made by closed labs, then open labs fast-follow” (which isn’t implausible given the documented distillation of Fable), and further that open labs cannot make make meaningful progress vs the closed labs except by fast-following and that they won’t pick up the ability to make progress after the closed labs are gone, then driving closed labs out of business slows down overall progress.
I think it's pretty hard to hold that worldview: Anthropic couldn't ship a reasoning model until they copied DeepSeek R1's homework, and they've all copied DS-style super-sparse MoEs at this point too.
That’s a really good point. Folks really need to read the papers coming out of these Chinese labs. Every paper from the DeepSeek team has been a step change.
With slightly different cherry-picking, you could equally well claim that DeepSeek couldn't ship a reasoning model until they copied the idea from OpenAI's o1-preview, and they also copied MoEs from Google Brain/Jagellonian University https://arxiv.org/abs/1701.06538 way back in 2017, too!
But ultimately these were ideas floating around in the air, if one group hadn't done the experiment, someone else would have.
the argument is that we should all fold and let sam altman burn trillions of dollars on naive scaling and pay monopoly prices for their closed APIs until the models are good enough to be closed off for "safety" reasons so that they can take an even larger cut by competing directly with us
I know it's a hard ask on this site, but I need you to start parsing content and not tone. It was a helpful bit of context, even if it was a bit vitriolic.
Because discourse around tone isn't productive. You could wipe this whole comment thread, starting with the parent of mine, and lose exactly zero information.
The content was 'you need to get your head checked'. That isn't tone, that directly implying that if you hold that position, there is something wrong with you. It's rude and unnecessary.
It’s a different topic but it’s correct imo. Open sourcing things is the best way to accelerate development.
Put another way, if you want to slow things down, put it behind a paywall, tag ideas ans “intellectual property” (meaning you’re the only one who can use it) and get the lawyers involved (injecting our slow legal system).
None of the above is a judgement call on whether development should be accelerated.
the only reason other labs can catch up is because the frontier labs can be distilled, and they siphon a % of the labs' revenue to reinvest into the next iteration
full accel would mean nationalizing the big 2 labs and locking in manhattan project style until RSI
(Edit: some great counterpoints in the replies. my view has definitely been changed!)
This K3 release just helped every other lab on the planet stay in the race by making it possible for them to build on top of it, placing them at the frontier starting line instead of having to spend billions of their own dollars and risking it all to attempt to catch up.
The open source contributions I linked to above will move the whole field forward and reduce the costs of training and inference for everyone.
Open science compounds on it self, every new advancement pushes the field forwards and opens up new grounds for future improvements.
It is impossible for a single closed lab to consistently stay ahead of the rest of the field, especially in a huge growing research area like machine learning. The only only advantage the big labs have is money, but the naive scaling game is not sustainable long term when you have to pay 10-100x more then the fast followers and we start getting more and more open models or use case specific models that can handle 90% of high volume use cases.
Research is a high variance, low expected value activity, meaning that the few large concentrated labs have to be conservative with their bets and double down on proven things when scaling up. The rest of the field is like a diversified portfolio, with thousands of players making smaller riskier bets that only require a few of them to succeed (like K3 did here, and DeepSeek a year ago)
EDIT: also if you look at most of the work from OpenAI, it's mostly taking existing promising open research work and scaling it up. (except for things like CLIP and etc from Alec Radford)
Appreciate this point. Back in grad school, I published a peer reviewed paper with all the source code and datasets used. Got heckled at a conference talk by staff from a commercial lab. They shouted, we figured this out five years ago, lol. Also our approach is still better. But they don’t release their source code or publish much, so no one knows what this approach is or if it’s actually better.
And years down the line, lots of other research labs used my code and cited my paper.
Oh this definitely happens all the time. I was an early employee at Clarifai, which won imagenet a year after alexnet and we were able to stay on the frontier for about 2 years before a bunch of open source models were matching our results. It was always some random PhD research project spinout or some random kid in Boston named Alec Radford.
We had a bunch of things that we never published that ended up being major research findings years later at top conferences.
being able to replicate it in the open means that there's nothing special about frontier models.
Frontier models would have to do something extraordinary or unique, or unreplicatable, because clearly there is no moat, and US companies are sitting on huge nvidia valuations and get surprised when competitors beat them.
OpenAI's head of strategic futures publicly stated that you can't explain the quality of the newest Kimi via distillation.
Further, you can just read the papers released alongside most open models. Plenty of hugely influential research results published that drive the frontier forward. It's not like these models are just existing architectures downloaded from Huggingface and trained on frontier lab APIs.
> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
+ the existing 100 million monthly active users, or more than 20 million US dollars for commercial products have to name Kimi clause
Yep. Cursor’s Composer 2 model is a good example, though it is not clear if they entered into an agreement with Moonshot before they got found out in March this year [1] or after.
Software licenses aren't enforced through litigation as much as they are enforced through the _threat_ of litigation and legal risk. In other words, pretty much every company pays lawyers to minimize legal risk. Those lawyers inevitably look at all the contracts, agreements, and software licenses, and tell the C-suite what to do in order to keep their legal exposure as low as possible. "Don't violate other companies' IP," is pretty low-hanging fruit in those conversations.
It is a very rare (and ballsy, and perhaps incompetent) company that ignores their lawyers' recommendations to adhere to the letter of all of the software licenses they are bound to.
The models they released in 2025 - https://huggingface.co/moonshotai/models - were clean MIT. They started doing the "modified MIT" thing in January 2026 with moonshotai/Kimi-K2-Thinking
To be clear, the restriction on "Model as a Service" past $20M yearly revenue is new to K3. K2.x had the attribution requirement for any commercial use with more than $20M monthly revenue.
In what way was the license "janky"? Moonshot isn't a trillion dollar US behemoth. If these terms allow it to release near frontier models with open weights, so startups and anyone with enough hardware can use them, while charging only companies with more than $20 million in revenue or 100 million users, that seems like a reasonable trade off.
So it's more like your personal definition, not the normal meaning of the word.
Ok, because "janky" is informal slang also meaning poor quality, unreliable etc. I understand that you're using janky as shorthand for not OSI compliant. I just think that can blur two separate issues: whether a license qualifies as open source under the OSI definition, and whether the license itself is unreasonable/badly designed.
Hah, so much for "open weights" :D Fair enough, they're at least downloadable, shame they didn't end up being actually open, nor open source, the community had really high hopes for this. But again, still available for download, so better than nothing else I suppose.
No, I don't know a single project I'd call "open source" (as understood by the FOSS community) that restricts what you can do with it, that'd make it very much "not open source" as you're discriminating against specific persons/groups/fields of endeavor.
Beyond reciprocity which is the entire point of that license, what restrictions does it come with? I guess you could say that it has a restriction of adding new restrictions, but surely that's not what you're talking about?
Copyleft licenses have restrictions. Now, those restrictions are definitely in promotion of the normative ethics that the FOSS community holds, but they are nonetheless restrictions.
It's worth noting that the weights of machine learning models are not subject to copyright in the United States as they are the product of an automated optimization process (e.g., stochastic gradient descent, expectation maximization, genetic algorithms) rather than human authorship. Granted, this has yet to be fully tested in court and going to court is expensive, so it's likely that your employer would prefer to err on the side of caution and respect such attempts at model licensing anyway. Nonetheless, this was partially tested last year in Thaler v. Perlmutter which affirmed that copyright requires human authorship, reading the Copyright Act's provisions on ownership, duration, and transferability as presupposing a human author[1].
If you want to assess the position of the U.S. Copyright Office for yourself, the relevant text can be found in the Compendium of U.S. Copyright Office Practices § 313.2, "Works That Lack Human Authorship"[2], which states:
> […] the Copyright Act protects “original works of authorship.” 17 U.S.C. § 102(a) (emphasis added). To qualify as a work of “authorship” a work must be created by a human being. See Burrow-Giles Lithographic Co., 111 U.S. at 58. Works that do not satisfy this requirement are not copyrightable.
> […] the Office will not register works produced by a machine or mere mechanical process that operates randomly or automatically without any creative input or intervention from a human author. The crucial question is “whether the ‘work’ is basically one of human authorship, with the computer [or other device] merely being an assisting instrument, or whether the traditional elements of authorship in the work (literary, artistic, or musical expression or elements of selection, arrangement, etc.) were actually conceived and executed not by man but by a machine.” U.S. COPYRIGHT OFFICE, REPORT TO THE LIBRARIAN OF CONGRESS BY THE REGISTER OF COPYRIGHTS 5 (1965).
Oh, and there's also a bit in the following Section 313.3, "Works That Do Not Constitute Copyrightable Subject Matter"[2], which explicitly excludes mathematical principles, formulas, algorithms, and equations, along with DNA sequences and other genetic or chemical compounds, regardless of whether they are produced by humans or by nature. If one takes the perspective that machine learning models are algorithms, the conclusions on copyrightability are pretty clear.
From the paper (page 6 with a comparison to GLU and SwiGLU) they are not using tanh directly (i.e. f(x) = tanh(x)) but:
f_gate(b,x) = b * tanh(x / b) * sigmoid(x)
f_up(b,x) = b * tanh(x / b)
Looking at the graph I wonder if this is to try and get the best of both GLU (better representation at higher values of x >~ 5) and SwiGLU (the value bump just before 0).
In the linked "Kimi-K3 Technical Report [pdf]" paper, section 2.3 (Stable LatentMoE, p6) has the table with those equations on (top of p7, using β_1 for the gate branch and β_2 for the up branch). They talk specifically about the function in section 2.3.2 (Sigmoid Tanh Unit GLU, bottom of p7).
Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question.
Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.
knowledge graph guided task synthesis. very cool! i have long wondered about the "how do you get good coverage of all the tasks" problem.
maybe some interesting theoretical work there around the rate of production of new knowledge itself and various mechanisms (human approaches, mechanistic approaches, etc).
Can someone explain what are teachers in Multi-Teacher On-Policy Distillation? I can imagine math, coding and other verifiable domains, but they also have biology? Is it where distillation from bigger models come in?
If I understand correctly, it's distillation via having a teacher model score each of the student's tokens for a problem based on their own probabilities of generating that token at each step in the sequence. The reward/loss is then applied as RL.
The multi-teacher bit seems to imply they're distilling from multiple models. It's light on the details, but it seems like it could be part of distilling from frontier/closed models. Provided they calculate the logprobs, which OpenAI seems to allow via API but not Anthropic. Maybe they have a way of estimating the logprobs externally?
This method can be used to learn any domain from the teacher. Biology included.
It's not that light on the details. I read the paper and they train several different models in parallel over a few different domains and then they distill from their own models to get the final model.
This is amazing to witness.
Moonshot open sourcing Kimi K3, a frontier AI and other components really means we are getting abundant AI for all of humanity.
Kudos to Moonshot for truly being what OpenAI should have been.
Fable-level and frontier AI should be open source and available to everyone for free.
> Kudos to Moonshot for truly being what OpenAI should have been.
Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license.
> Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license.
They will probably never release the training data because that represents a large part of their competitive moat. The same is true of US companies (Google, OpenAI, Meta etc) none of which has released the full training data for its open models.
They use private datasets that cost a lot to acquire, synthetic datasets and a lot copyrighted material for which they don't have licensing.
What matters most is that, with the necessary hardware, I can download a near frontier model, run it and modify it however I want. The other concerns you mentioned are just noise. And if my company is generating $20 million in revenue or serving 100 million users, it can probably afford a relatively inexpensive commercial license.
>They will probably never release the training data because that represents a large part of their competitive moat
Not that what you've written isn't the case. However, in addition to what you've written, (or probably even before what you've written), there's the fact that everything they're training on is stolen IP. Same with US LLM labs.
Let's not kid ourselve's about where the training data is coming from. They are not asking artists, writers, coders, content creators, etc etc etc for permission to use their creations.
Anthropic, Moonshot et al are doing incredible things, but we shouldn't gloss over the costs. Both present and future costs are kind of enormous.
> They use private datasets that cost a lot to acquire, synthetic datasets and a lot copyrighted material for which they don't have licensing.
Yeaaah, and this, of course, is worth it, because it leads to you being able to download a near frontier model. Don't get me wrong, long-term humanity is probably better of with science with little regards to pesky things like ethics and provenance, but we also have a tendency to not fully realize the downstream or wider effects until way too late.
And sure, there is a lot of reasons to go with keeping your software proprietary too, I'm not trying to claim otherwise, same with training data. It's just that usually we don't call proprietary software "open source" unless it is open source, regardless of the reasons someone keep it proprietary or not, could be for whatever reason really.
Exactly. I wish people would just stop saying "open source" regarding models. Even "open weight" is disingenuous. "self-hostable" would be more honest.
It's a gradient almost, with some steps. So far, I think you could categorize every single released so far as one of:
- Proprietary - No access beyond remote endpoints
- Downloadable - You can run it, but there are restrictions and training data/code isn't public and/or under FOSS license, nor are the weights under a FOSS license
- Open weights - The weights are under a FOSS license and downloadable without restrictions, but not all of training code/data is public or under a FOSS license
- Open source - The model architecture, weights, training code and data are all under a FOSS license, and you can freely download and use them without any restrictions
Few models hit the mark for "open source model", many models people call "open weights" would go under "downloadable" here, which personally I think would be accurate.
"Downloadable" should be "weights available" by analogy to "source available". K3 is weights available because it includes usage restrictions. K2.6 is open weights, because the only restriction is a weaker version of the "obnoxious BSD advertising clause", which is accepted as FOSS.
> we build a self-evolving, hierarchically organized knowledge graph that agents continuously expand through web-scale exploration across knowledge-intensive and coding domains
Latest "big" release from any of the bigger American lab must have been GPT-OSS-120b I think? Released ~summer 2025, so pretty much one years ago. Doesn't seem like it'll happen by itself, so something either forcing their hand figuratively, or something forcing their hand literally.
Personally I was wishing/hoping for one of the recent Gemma releases to be in the ~100B class at least, but sadly Google is keeping that all for themselves.
NVIDIA-Nemotron-3-Ultra-550B-A55B was released in June 4th 2026 and I think it was the largest open US model until thinking machines' Inkling (975B) was released a couple of weeks ago.
if it's to be believed, it's so easy. you distill it, and you have a copy in 2 weeks, isn't that what China is doing? so 2 weeks from now, we should have one.
Your best bet will be to start a network optimized ec2 (or gcp compute instance) and save to some bucket. Cloud network will be a lot more stable than home.
Back of the envelope calculation (could be off, correct me if I am)
If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s.
Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper!
As long as a company believes that the openweight models will continue to get more capable and 'AI is here to stay', this model provides the first solid footing for a decision to just buy a rack.
As far as I can tell, license fees are only applicable if you have more than 20m USD/year revenue from services provided using the model, or serve more than 100m users.
I think internal use is allowed at any scale in the license?
> 4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; [...]
It's very hard to make sure none of the outputs are ever made available to third parties. Source code can end up widely distributed (e.g. client-side js, open source). Prose will frequently get shared across organization boundaries (e.g. emails, websites, documents).
License for LLMs already have very little strength (it's not clear at all that they have any legal basis whatsoever) an AI lab is very, very unlikely to sue you for violating this kind of license clause for the reason you mentioned. It's clearly worded in a way to deter people from running a third party inference business out of it, they would have been much more explicit if they wanted to deter people from running it for internal use.
40% overhead is quite typical, so you'd be looking at $120k/year. In the USA I'd consider that a competitive salary for an admin capable or keeping a $6M rack of specialized hardware running 24/7.
Yeah but you don't need two such people, or even one, dedicated to this single rack.
A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.
My company is not even the same ballpark, but even we already have people for it. And doesn't include the fact that you can get a colo location and just a MSP or a contractor to do it for you
Understood but sharing your existing devops resources with this will soon become a bottleneck especially when any major downtime will keep several engineers (and long-running agents) blocked from any meaningful work until availability improves.
Not my experience, from an SMB that maintains its own hardware and services. You have a certain contingent of competent engineers who distribute their work across projects, and it generally works out fine. Or course you plan with some redundancy and fall-back plans in your systems.
You’ll slap some training on existing technologists/infra/sysadmin folks and perhaps have a support contract for the edge cases (hardware troubleshooting and advanced replacement).
(managed an entire data center building with thousands of servers a lifetime ago with ~2-3 other people, it’s only gotten easier over the last two decades imho)
As mentioned, it is a "large enough company" already; they have full time sysadmins running things.
Adding another rack beside the VMWare cluster, managing any storage/networking issues, etc. will be incremental costs; they already have a pager (probably not a pager not anymore just an app on their phone) like rotation schedule etc.
> I don’t trust hosted LLMs for anything that needs to be private
I'd update this to
'I don’t LLMs for anything that needs to be private'
What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.
Why do you feel that creating the LLM from scratch is sufficient to trust it? Are you suggesting that you personally would read all 15 trillion tokens (plus every single agentic trade used in RL, along with its relative advantage in the batch) and personally guarantee that gradient descent would train a model which would not exfiltrate your corporate data?
Or that perhaps you have a perfect alignment algorithm which you are unwilling to share with the broader research community (evil)?
You also need a place to put it. With liquid cooling and extremely dense power capability. Your typical colo or closet server room isn't going to cut it.
Not GP, but my educated guess is that they are running a system with between 4 and 6 MI325 or MI355x or similar AMD GPUs. With the 50k tps as the total figure for all parallel requests. Those cards have a lot of memory for their price, allowing you to push to really high batch sizes while still having a large context size for each request
Are you developing software? Is most of it used on a coding agent? (Like Claude Code or ChatGPT Codex?) If so, what coding agent do you use? If you're not developing software what do you use it for (roughly)?
1e21 flops is hilariously wrong. for reference the llama 3 8B model (https://arxiv.org/pdf/2407.21783) used 10 times that many flops. This model is 350x bigger in total params and 12x bigger in active params and was trained on 3x the data.
That's pretty impressive, I don't want to be the one undermine but I personally think the industry is shifting to rather designs that enable Models to work instead of chugging it all to data and compute ?
m00dy | 5 hours ago
End-to-end tokens/sec and cost on realistic coding agent trajectories, including tool outputs and retries, not isolated decode benchmarks.
Cache hit rates and prefill cost for branching, multi-turn sessions.
Router-load distributions after post-training, where expert collapse or specialization problems often show up.
storus | 5 hours ago
whimsicalism | 5 hours ago
m_ke | 5 hours ago
Anyone who claims open source and open weights models are "decel" needs to get their head checked
https://github.com/MoonshotAI/MoonEP
https://github.com/kvcache-ai/AgentEnv
https://github.com/MoonshotAI/FlashKDA
jvanderbot | 5 hours ago
ike_a | 5 hours ago
Decel:
- Potentially reduces investor appetite for funding big labs.
- More risk of powerful AI getting in bad hands -> more regulation.
Accel:
- More competition so big labs can't rest on laurels.
- More research in open, so all labs can accrete advancements faster.
I feel like open-source = acceleration has a much more clear argument. (and how bad would deceleration be in any case?)
Iolaum | 5 hours ago
StevenWaterman | 5 hours ago
f311a | 5 hours ago
I think you meant less research and experiments in big labs because they don't get all the AI money.
Training is expensive, but they also have more than 10 000 of employees combined and they cost a lot of money.
zozbot234 | 5 hours ago
Kimi K3 is plausibly a lot less dangerous than a totally jailbroken ChatGPT/Gemini/Claude Sonnet (let alone Opus or Fable!) and it's quite deeply weird how no one seems to be calling for those models to be banned or restrained by further regulation. Why the double standard against the less concerning (but more efficient!) open weight models?
ike_a | 4 hours ago
Is there a world where open source models end up at the frontier, or do you think there are structural/first-principles reasons why this won't happen?
zozbot234 | 4 hours ago
sosodev | 4 hours ago
The problem with the decel/accel rhetoric is that it lacks nuance.
Smaug123 | 4 hours ago
reissbaker | 4 hours ago
chorizo | 4 hours ago
yorwba | 2 hours ago
But ultimately these were ideas floating around in the air, if one group hadn't done the experiment, someone else would have.
m_ke | 3 hours ago
acedTrex | 5 hours ago
viccis | 4 hours ago
Almondsetat | 4 hours ago
viccis | 4 hours ago
SubiculumCode | 4 hours ago
jvanderbot | 3 hours ago
BeetleB | 3 hours ago
I, for example, dislike reading comments complaining the submission (or another comment) is LLM generated. Focus on the content, not the style.
I'm not going to have my way, and nor shall you.
SirLordBoss | 4 hours ago
janalsncm | 3 hours ago
Put another way, if you want to slow things down, put it behind a paywall, tag ideas ans “intellectual property” (meaning you’re the only one who can use it) and get the lawyers involved (injecting our slow legal system).
None of the above is a judgement call on whether development should be accelerated.
vanuatu | 4 hours ago
the only reason other labs can catch up is because the frontier labs can be distilled, and they siphon a % of the labs' revenue to reinvest into the next iteration
full accel would mean nationalizing the big 2 labs and locking in manhattan project style until RSI
(Edit: some great counterpoints in the replies. my view has definitely been changed!)
m_ke | 4 hours ago
There's no chance K3 is a distill of Fable, it came out way too soon after the limited fable release to be feasbile.
If you look at all of the top ML conferences, chinese labs contribute way more to advances in ML than "Open"AI and Anthropic: https://www.reddit.com/r/TheMachineGod/comments/1pi4q7f/pape...
This K3 release just helped every other lab on the planet stay in the race by making it possible for them to build on top of it, placing them at the frontier starting line instead of having to spend billions of their own dollars and risking it all to attempt to catch up.
The open source contributions I linked to above will move the whole field forward and reduce the costs of training and inference for everyone.
Open science compounds on it self, every new advancement pushes the field forwards and opens up new grounds for future improvements.
It is impossible for a single closed lab to consistently stay ahead of the rest of the field, especially in a huge growing research area like machine learning. The only only advantage the big labs have is money, but the naive scaling game is not sustainable long term when you have to pay 10-100x more then the fast followers and we start getting more and more open models or use case specific models that can handle 90% of high volume use cases.
Research is a high variance, low expected value activity, meaning that the few large concentrated labs have to be conservative with their bets and double down on proven things when scaling up. The rest of the field is like a diversified portfolio, with thousands of players making smaller riskier bets that only require a few of them to succeed (like K3 did here, and DeepSeek a year ago)
EDIT: also if you look at most of the work from OpenAI, it's mostly taking existing promising open research work and scaling it up. (except for things like CLIP and etc from Alec Radford)
chorizo | 4 hours ago
And years down the line, lots of other research labs used my code and cited my paper.
m_ke | 4 hours ago
We had a bunch of things that we never published that ended up being major research findings years later at top conferences.
pluto_modadic | 4 hours ago
Frontier models would have to do something extraordinary or unique, or unreplicatable, because clearly there is no moat, and US companies are sitting on huge nvidia valuations and get surprised when competitors beat them.
ZeroGravitas | 4 hours ago
calebkaiser | 3 hours ago
Further, you can just read the papers released alongside most open models. Plenty of hugely influential research results published that drive the frontier forward. It's not like these models are just existing architectures downloaded from Huggingface and trained on frontier lab APIs.
fahrradflucht | 5 hours ago
> If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 20 million US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must enter into a separate agreement with Moonshot AI before using the Software or its derivative works for any commercial purpose.
+ the existing 100 million monthly active users, or more than 20 million US dollars for commercial products have to name Kimi clause
throwaway27448 | 5 hours ago
ffsm8 | 5 hours ago
Iolaum | 5 hours ago
JumpCrisscross | 5 hours ago
What was it?
drawnwren | 5 hours ago
1 - https://x.com/Kimi_Moonshot/status/2035074972943831491?lang=...
paxys | 5 hours ago
throwaway27448 | 2 hours ago
amazingamazing | an hour ago
ayewo | an hour ago
1: https://x.com/fynnso/status/2034706304875602030
bityard | 4 hours ago
It is a very rare (and ballsy, and perhaps incompetent) company that ignores their lawyers' recommendations to adhere to the letter of all of the software licenses they are bound to.
mike_hearn | 3 hours ago
simonw | 5 hours ago
The models they released in 2025 - https://huggingface.co/moonshotai/models - were clean MIT. They started doing the "modified MIT" thing in January 2026 with moonshotai/Kimi-K2-Thinking
zozbot234 | 5 hours ago
simonw | 4 hours ago
lossolo | 4 hours ago
bilbo0s | 4 hours ago
For people at the top of the tech pyramid, I could totally see why they'd want to put an end to all this "Open Source LLM" stuff.
simonw | 4 hours ago
"Modified MIT" is the perfect example of that.
I'm not saying it's unreasonable, or that you can't release under such a license - it's your software, use whatever license you like!
I'll call it "janky" when you do.
lossolo | 4 hours ago
simonw | 4 hours ago
embedding-shape | 5 hours ago
jhonof | 4 hours ago
embedding-shape | 4 hours ago
parodysbird | 4 hours ago
embedding-shape | 4 hours ago
parodysbird | an hour ago
ericpauley | 4 hours ago
cosmojg | 2 hours ago
cosmojg | 3 hours ago
If you want to assess the position of the U.S. Copyright Office for yourself, the relevant text can be found in the Compendium of U.S. Copyright Office Practices § 313.2, "Works That Lack Human Authorship"[2], which states:
> […] the Copyright Act protects “original works of authorship.” 17 U.S.C. § 102(a) (emphasis added). To qualify as a work of “authorship” a work must be created by a human being. See Burrow-Giles Lithographic Co., 111 U.S. at 58. Works that do not satisfy this requirement are not copyrightable.
> […] the Office will not register works produced by a machine or mere mechanical process that operates randomly or automatically without any creative input or intervention from a human author. The crucial question is “whether the ‘work’ is basically one of human authorship, with the computer [or other device] merely being an assisting instrument, or whether the traditional elements of authorship in the work (literary, artistic, or musical expression or elements of selection, arrangement, etc.) were actually conceived and executed not by man but by a machine.” U.S. COPYRIGHT OFFICE, REPORT TO THE LIBRARIAN OF CONGRESS BY THE REGISTER OF COPYRIGHTS 5 (1965).
Oh, and there's also a bit in the following Section 313.3, "Works That Do Not Constitute Copyrightable Subject Matter"[2], which explicitly excludes mathematical principles, formulas, algorithms, and equations, along with DNA sequences and other genetic or chemical compounds, regardless of whether they are produced by humans or by nature. If one takes the perspective that machine learning models are algorithms, the conclusions on copyrightability are pretty clear.
[1] https://media.cadc.uscourts.gov/opinions/docs/2025/03/23-523...
[2] https://www.copyright.gov/comp3/chap300/ch300-copyrightable-...
whimsicalism | 5 hours ago
pinkmuffinere | 4 hours ago
rhdunn | 3 hours ago
whimsicalism | 3 hours ago
lostmsu | 2 hours ago
rhdunn | an hour ago
Figure 2 (p3) has the architecture diagram for K3 (which is the same one on the Kimi K3 blog post https://www.kimi.com/blog/kimi-k3). AFAICT, that diagram along with section 2.3 should answer your question.
Note: I'm not familiar with LLM/NN architecture to answer it more precisely than that; I only have a surface level understanding from watching various YouTube channels like 3B1B and Welch Labs on the subject.
a-dub | 5 hours ago
maybe some interesting theoretical work there around the rate of production of new knowledge itself and various mechanisms (human approaches, mechanistic approaches, etc).
eamag | 5 hours ago
porridgeraisin | 4 hours ago
sosodev | 4 hours ago
If I understand correctly, it's distillation via having a teacher model score each of the student's tokens for a problem based on their own probabilities of generating that token at each step in the sequence. The reward/loss is then applied as RL.
The multi-teacher bit seems to imply they're distilling from multiple models. It's light on the details, but it seems like it could be part of distilling from frontier/closed models. Provided they calculate the logprobs, which OpenAI seems to allow via API but not Anthropic. Maybe they have a way of estimating the logprobs externally?
This method can be used to learn any domain from the teacher. Biology included.
mike_hearn | 3 hours ago
htrp | an hour ago
Does openai still allow logprobs in their current gen models?
colesantiago | 5 hours ago
Kudos to Moonshot for truly being what OpenAI should have been.
Fable-level and frontier AI should be open source and available to everyone for free.
embedding-shape | 4 hours ago
Kudos to Moonshot for making these weights available for download. Lets not fool ourselves and claim these are "open source" by any understanding of the concept though, there are usage restrictions (even if you download them) and also training data isn't clearly broken down either, nor it it actually using a FOSS license.
lossolo | 4 hours ago
They will probably never release the training data because that represents a large part of their competitive moat. The same is true of US companies (Google, OpenAI, Meta etc) none of which has released the full training data for its open models.
They use private datasets that cost a lot to acquire, synthetic datasets and a lot copyrighted material for which they don't have licensing.
What matters most is that, with the necessary hardware, I can download a near frontier model, run it and modify it however I want. The other concerns you mentioned are just noise. And if my company is generating $20 million in revenue or serving 100 million users, it can probably afford a relatively inexpensive commercial license.
bilbo0s | 4 hours ago
Not that what you've written isn't the case. However, in addition to what you've written, (or probably even before what you've written), there's the fact that everything they're training on is stolen IP. Same with US LLM labs.
Let's not kid ourselve's about where the training data is coming from. They are not asking artists, writers, coders, content creators, etc etc etc for permission to use their creations.
Anthropic, Moonshot et al are doing incredible things, but we shouldn't gloss over the costs. Both present and future costs are kind of enormous.
embedding-shape | 4 hours ago
Yeaaah, and this, of course, is worth it, because it leads to you being able to download a near frontier model. Don't get me wrong, long-term humanity is probably better of with science with little regards to pesky things like ethics and provenance, but we also have a tendency to not fully realize the downstream or wider effects until way too late.
And sure, there is a lot of reasons to go with keeping your software proprietary too, I'm not trying to claim otherwise, same with training data. It's just that usually we don't call proprietary software "open source" unless it is open source, regardless of the reasons someone keep it proprietary or not, could be for whatever reason really.
dboreham | 4 hours ago
embedding-shape | 4 hours ago
- Proprietary - No access beyond remote endpoints
- Downloadable - You can run it, but there are restrictions and training data/code isn't public and/or under FOSS license, nor are the weights under a FOSS license
- Open weights - The weights are under a FOSS license and downloadable without restrictions, but not all of training code/data is public or under a FOSS license
- Open source - The model architecture, weights, training code and data are all under a FOSS license, and you can freely download and use them without any restrictions
Few models hit the mark for "open source model", many models people call "open weights" would go under "downloadable" here, which personally I think would be accurate.
mrob | 2 hours ago
embedding-shape | 2 hours ago
eamag | 5 hours ago
That's interesting!
lenerdenator | 5 hours ago
boomskats | 5 hours ago
chrsw | 5 hours ago
embedding-shape | 5 hours ago
Personally I was wishing/hoping for one of the recent Gemma releases to be in the ~100B class at least, but sadly Google is keeping that all for themselves.
layer8 | 4 hours ago
embedding-shape | 4 hours ago
lossolo | 4 hours ago
https://thinkingmachines.ai/news/introducing-inkling/
Stagnant | 4 hours ago
whimsicalism | 4 hours ago
embedding-shape | 4 hours ago
Deeply fun "care about improving people's lives" quote on your user page :p
whimsicalism | 3 hours ago
segmondy | 4 hours ago
0x000xca0xfe | 3 hours ago
RazorBucksICO | 2 hours ago
brcmthrowaway | 4 hours ago
weberer | 4 hours ago
se4u | 2 hours ago
GodelNumbering | 4 hours ago
If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s.
Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper!
As long as a company believes that the openweight models will continue to get more capable and 'AI is here to stay', this model provides the first solid footing for a decision to just buy a rack.
reckless | 4 hours ago
vidarh | 4 hours ago
petu | 4 hours ago
> 4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; [...]
jsnell | 3 hours ago
stymaar | 2 hours ago
999900000999 | 4 hours ago
That another 400 to 700k.
It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.
clint | 4 hours ago
russell_h | 4 hours ago
Why not? Do you trust AWS with things that need to be private?
Kevcmk | 4 hours ago
senderista | 4 hours ago
wongarsu | 4 hours ago
arjie | 4 hours ago
LeonM | 4 hours ago
40% overhead is quite typical, so you'd be looking at $120k/year. In the USA I'd consider that a competitive salary for an admin capable or keeping a $6M rack of specialized hardware running 24/7.
margalabargala | 3 hours ago
A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.
999900000999 | 3 hours ago
Who is going to fix it when the api does something weird ?
Who is going to proactively make sure it’s not overheating?
Chat GPT has enterprise contracts for a reason.
stymaar | 2 hours ago
> A company of the size that this is worthwhile for, probably has dedicated devops on staff already
margalabargala | 2 hours ago
I didn't say it's fire-and-forget. I'm saying all that is maybe a day of work every 3 months.
niltecedu | 40 minutes ago
GodelNumbering | 4 hours ago
Not a devops but I'd say one full time is already too many.
dboreham | 4 hours ago
layer8 | 4 hours ago
ayewo | 3 hours ago
layer8 | 3 hours ago
ayewo | an hour ago
Add to this number another $1.5m/yr in opex, so not sure I’d call such an enterprise wealthy enough to spend those kinds of sums on LLMs an “SMB”.
JacobAsmuth | an hour ago
lumost | 4 hours ago
Having these models in the open caps the inference margin.
slicktux | 4 hours ago
toomuchtodo | 4 hours ago
(managed an entire data center building with thousands of servers a lifetime ago with ~2-3 other people, it’s only gotten easier over the last two decades imho)
shrubble | an hour ago
Adding another rack beside the VMWare cluster, managing any storage/networking issues, etc. will be incremental costs; they already have a pager (probably not a pager not anymore just an app on their phone) like rotation schedule etc.
w0m | an hour ago
I'd update this to
'I don’t LLMs for anything that needs to be private'
What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.
mdp2021 | an hour ago
A NN per se is a file... The executable that runs it can "act"...
JacobAsmuth | an hour ago
Or that perhaps you have a perfect alignment algorithm which you are unwilling to share with the broader research community (evil)?
JimmaDaRustla | 49 minutes ago
kcb | 3 hours ago
FuriouslyAdrift | 2 hours ago
We've already saved money compared to last years token cost on Claude/Gemini
mdp2021 | an hour ago
Equipped with what? Is it CPU based inference, a mix...?
wongarsu | 21 minutes ago
logicallee | an hour ago
freediddy | an hour ago
ajju | 58 minutes ago
Tejush | 4 hours ago
lostmsu | 2 hours ago
JacobAsmuth | an hour ago
NguyenDat377 | an hour ago
Mohiuddin7 | an hour ago
htrp | an hour ago
logicallee | an hour ago