My current feedback on Inkling 2 Small: 1. it feels intelligent, and can solve open problems throw at it like other quasi-frontier models (DeepSeek 4 Flash 0731 / DeepSeek 4.1 Flash 1017 / GLM 5.3 Flash etc). 2. its responses sometimes can be odd (they are honest about it in the blog post!), can have wrong use of markdown syntax, sometimes overly brief, etc. 3. tool uses are good, but sometimes have some old take on tools (too quickly to assume image generation tools cannot do text etc).
EDIT: by "open problems", I meant the problems I don't really formulate how to solve it first in my mind and then throw to it. I just describe what I want and let it to figure it out.
In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.
Let's hope these make it into master, and don't end up like the diffusiongemma patchsets. It is always a bit sad if the exotic and exciting models don't make it into master and all the people using prebuilt llama can't just give them a try. Tho I can understand why these in particular might not be wanted in master.
Wow, that coding demo was super impressive. Can't wait to try it out. Just need to wait for the hardware market to cool down a bit so we can run these on our own commodity hardware.
It's just a ryzen 395 ai max box with 128GB but that I was lucky to buy before the ramagedon happened. It just happens to run a fair bit of open source models decently and the models are getting better. I got it for 2100 EUR and now it looks like a steal.
DSF 4.1 and Qwen 3.8 27b. I have to switch linux distro though been on Bazzite and the Fedora Atomic underpinnings is just killing me for this stuff just makes it harder than it needs to be to install stuff.
Indeed. I 've been using Sage A1.5 35b-A3b to have more speed because of strix halo limited memory bandwidth. Looking forward to testing Inkling-2-mini once it has lamma.cpp support. Genuinely excited from a model outside Qwen3.6-35B-A3B ecosystem competing in that space.
P.S. Qwen3.8-Flash-Next is also a very good model for Srix-Halo/128Gb Framework Desktop but I liked to have more RAM for other things as well even when I was running it in Q4. Usable though.
This is bonkers good if these benchmarks hold up in real-world scenarios. To be at the level of Qwen3.8-27B with a looped transformer model is quite the flex. They just left this note on reddit:
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
Goes a couple or three times through the same weights in a middle layer of the model. Nemotron's Dory pioneered this I think. Seems to give better results with smaller models.
I believe someone found out (DIY style) that looping a couple of layers (found through trial and error) in ordinary LLM's yielded some benefits quite a while ago. it was a surprise as there was no looping during training.
then there was a fully looped model (forget the name) where all the layers were looped (which is an obvious mistake as it's known LLM's have token processing and reasoning stages) and showed some decent results.
and now the concept worked itself into the mainstream.
There's a lot of interesting work around this going back a little while. It's spiritually inline with the thinking around RNNs, which date back further. The Universal Transformer (2018) is the first research paper I'm familiar with that really pushed the idea as applied to transformer models, but I'm sure there are others (as is the case with everything in deep learning, someone tried it 8 years before you using different nomenclature).
It's a pretty broad bucket that contains a few different research lineages, but the gist (when talking about Transformers) is that instead of stacking 12 latent blocks, you can stack 4 and let them loop 3 times.
That's just a general simplification though. One of the key motivators in this general area of research is adaptive latent reasoning, meaning in theory, some tasks only require 2 loops and some require 8.
There's a lot of different approaches here and many of them approach the problem from different traditions. Some looped transformers use a fixed loop count, others follow after deep equilibrium models and iterate towards a fixed point or some other stopping mechanism.
Some fun reading if you're interested in this field:
- TinyRecursiveModels - Blew up a couple of years ago when it performed really well on ARC AGI 1 using a comparatively small model with a large task-level table of embedding prefixes. https://github.com/samsungsailmontreal/tinyrecursivemodels
If not this model it's gonna be one of the whole slew on the way, especially once we start seeing looped diffusion (and similar) models. They're gonna smash your expectations.
Every time I look at these I think, yep, another small model claims to outperform Muse Glimmer.
And then I test it in comparison and I remember Goodhart's Law.
This model, for sure, is interesting. But there's more than one way to peel a potato, and again and again I find that in practice these benchmarks don't really seem to mirror normal "coding assistant" uses.
People have got really hooked on benchmarks and token speeds, but a fast-outputting model that wastes endless tokens on thinking and then gets it wrong is not much use.
And no amount of tool-calling search really seems to "close the knowledge gap" when a small model is absolutely insisting something exists when it does not.
When inkling first released I tested it out and thought it was the best prose from an LLM in my quick sampling -- easy-reading and easy to understand. Big hopes I can write code with one of these models one day.
No. It's either some router that sends the query to the best possible model or just a Chinese (possibly bench-maxed) model. US models are never stealth.
Very interesting to see them publish a looped transformer. It's been reported that OpenAI are now using looped transformers, though of course not confirmed.
I think this may be the first public example of a looped transformer model good enough for real world use cases?
Going to test this right away on my AMD 7900XTX and see how it does compared to Qwen 3.8 27B (or DaisyRose 27B that I'm currently running). Given that Inkling 2 Mini is a MoE, it should provide way higher decode speed, but with comparable performance. Exiting!
I had astra patch in both required llama.cpp PRs and it's passing a few tests locally on my 7900XTX so far, but ~500 tok/s prefill and ~26 tok/s decode are sobering: The new three-time-looping might be cutting speeds. Maybe there's something here still though, maybe it needs less reasoning output for results to be usable/good? More testing needed!
edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE
edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :(
Hm, that seems strangely slow. Qwen3.8-27B/DaisyRose-27B Q4 currently run with ~66 t/s and ~800 t/s prefill with 3 slots and about 190k kv cache. It performs really well so far.
Yeah I've had 27B (dense model) up to similar speeds on this setup as well. I'll hold off to compare my speeds to others, maybe it's too early or it's particular to my setup.
Got it to run, but a few bumps in the road – mostly because this is a very fresh model.
Flash Attention (FA) is not supported on the current llama.cpp (at least not for ROCm) for this model so KV cache is forced to f16 for the value (V) and I cannot lower it to q4 go get decent KV cache. But the key (K) could be set to q4, and that got me about 100k of ctx. Current benching shows ~28 tok/s decode and ~481 tok/s prefill.
It's a start, but I expected better speeds even on a non ideal setup. I'll keep experimenting!
Hey, just in case.. I have been maintaining this fork of llama.cpp, I run also 7900XTX, 3 in my case... and I want to know if you see improved speed... notice the ngram mod where instead of loosing the ngrams they are stored in disk, I have a 31M slot and so far it's being working really nice for me
I'm seing around 2800t/s on pp at the very beginning of the conversation, and then going to around 800 by the 256k context size...
I've also update the value in the gguf to have instead of context size 256k be triple and have use it all the way to around 600k context size
I tried that for awhile too. Fun tip: You don't have to retrain the entire model after looping it to make it coherent again.. Just pick layers carefully then repair the residual by doing some parent distillation. ;)
That took me from needing to rent space in avast to just running it locally on my sparks and getting comparable results.
I also discovered that you can fix a lot of the damage caused by quantization the same way. Just do 200-500 rounds of parent student distillation with everything frozen but the residual, often the quantized version will improve drastically.
I think this model will be a crazy big deal, it might be the first model that can reliably run multi-agent reasoning harnesses on local hardware, as this is a MoE model and can have several agents each with their own KV caches that fit into VRAM.
Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.
> TML really are a bunch of rationalists who know what they're doing
yes
> sort of like Anthropic.
no. Anthropic are not "rationalists," they are absolute fear-mongers. from "no software developers in 6 months" to "this could enable bioterrorism so we do not even let you mention the word 'virus' in any context", I do not see a hint of rationality.
I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.
I'm guessing one of these local model companies will get picked by Apple sooner or later. It is just so within their business model and they have the hardware.
Apple refuses to pay ai engineers an ai engineer salary. They're not going to acquire one of these companies until the cost of issuing them is within their comfort zone.
Anyone following the news is aware that we've had Mythos-comparable cyber red-teaming capabilities for quite a few months now. The sky hasn't fallen, instead people have been using it for reverse engineering, postmortems, pentesting and decomp projects. In a world where frontier labs and governments can coordinate large-scale cyberattacks with AI, having an uncensored open cyber model is necessary for defending against them. The alternative is much worse.
But hey, you're welcome to increment your P(doom) until the year 2057 if it makes you feel better. Maybe the government will finally put their foot down with the release of Qwen 28, which according to OpenAnthropicTimeWarnerPepsico will have unprecedented puppy-kicking use cases that necessitate regulation.
Whoa, not just a model that acknowledges and embraces using a looped architecture (GPT Astra 6+ and Sol 6.1+ are only [heavily] rumored to be looped), but it's open weights, too?
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.
But, Inkling 2 Mini is the first open weight MoE looped transformer of any note that I've seen (the others are dense models), and the benchmark scores really do look incredible.
If it can outsmart Qwen3.8-27B while using substantially fewer active parameters... this is going to be really great.
I think that's really the key here right? It doesn't really "beat" Qwen3.8-27B.. more of like it's on par / a few points ahead on some, behind on others. But contrary to the Qwen (which is dense) it actually does it as an MoE - so less active parameters.
And the fact that they trained it as a looped model in 4-bit native is also kind of impressive to me as well, because it combines recurrence, routing, and narrow-format - all of which adversely impact numeric stability during training.
This has been my line of thought for a years. If we are aiming for on device intelligence, looping (or other methods of parameter reuse) are one of the best methods since consumer hardware is primarily memory constrained.
Which is counter to the way most open weight models have been leaning. Many current models are MoE (Mixture of Experts) which by their very nature are parameter heavy. They are trading parameter efficiency for compute efficiency which is kind of the wrong tradeoff for current consumer hardware like gaming GPUs.
Consumer hardware doesn't really boil down to a single constraint on how much memory is available, in my opinion.
Consumer hardware has limited memory, limited memory bandwidth, limited compute, and limited storage. Any model designed for consumer hardware needs to tackle all four, while still optimizing for intelligence.
MoEs save on both memory bandwidth and compute compared to a same-size dense model, and importantly, MoEs can still run quickly even with partial offloading. People have even demonstrated getting pretty good performance out of MoEs streaming from SSD with careful optimization.[0] Dense models fall apart if they aren't run entirely in the fastest memory pool.
Memory bandwidth is scarce on consumer laptops and smartphones. Looping demands even more memory bandwidth than running the same parameters once, so it is not an obvious win. If looping can make the model significantly more intelligent, it might be worthwhile. It is interesting to see a MoE combined with looping, as with Inkling 2 Mini. On your point about memory being the constraint, looping increases the size of Inkling 2 Mini's KV cache because the looped layers don't share KV between loops.
With DSpark-style specdec getting better and better, dense models make more sense than they used to since you can amortize the memory bandwidth better. The problem is then prefill (prompt processing), since specdec does nothing to help with that, and processing long inputs (search results, reading documents, etc) will still feel very slow. A MoE can breeze through that same prefill since it doesn't need as much compute.
With the introduction of Engrams/PLEs, model designers now have another axis for adding knowledge to models at the cost of storage, with comparatively little additional compute or memory bandwidth.
There are tradeoffs on everything. Despite my own extensive testing of different models on different hardware setups, I have not found any one-size-fits-all model architecture for consumer hardware. It is a hard problem. Small models are getting better all the time, and I think even smartphone-sized models will pass the threshold of usable intelligence for most people in the next year. Spark-X2.6-4B[1] and LFM3-1.7B[2] are surprisingly good for their sizes. Inkling 2 Mini looks great, supposedly offering Qwen3.8-27B's intelligence with much less compute required.
> A MoE can breeze through that same prefill since it doesn't need as much compute.
Important caveat: if not fully offloading[0] then the MoE model's prefill speed completely tanks, as prefill is heavily dependent FLOPS and parallelization and CPUs suck at that compared to GPUs.[1]
[0] 'Offload' is being used in the llamacpp sense of using the GPU instead of CPU
[1] Generally, prefill is bottlenecked by compute while generation is bottlenecked by memory bandwidth
Fortunately, that's not really how it works anymore.
When you're prefilling, you can afford to amortize the cost of sending the experts to a scratch space on the GPU then running a bunch of tokens through that set of experts, then repeating with the next set of experts, and so on. There's no reason to compute those on the CPU, and llama.cpp already does move prefill to the GPU by default for partially-offloaded MoEs.
Where performance would still tank is if you don't ensure the microbatch (ubatch) is large enough to amortize that cost, and if you're prefilling only a small input, it's not going to reach full speed. For a large input, you can think of each expert as visiting the GPU once for every ubatch of input tokens. A larger ubatch means fewer transfers.
Try increasing `-ub` (2048 is a sane default, but 4096 or higher can sometimes be desirable), and watch the partially-offloaded MoE prefill performance magically go way up on llama.cpp for large inputs.
EDIT: Also, if you haven't tried replacing `--n-cpu-moe` with `--cpu-moe` + `--moe-cache-mib` (with all of your GPU memory that you can spare), it is even better than `--n-cpu-moe`, since real world models don't distribute work over the experts in a perfectly random way. The moe-cache speeds up decode/output tokens per second.
> Wonder what it means for safety and interpretability at this level of availability.
We already know from official model cards that the GPT-6 models are substantially less monitorable than other (non-looped) models. Which is not surprising, since they are doing some amount of Neuralese thinking between each token.
What are some models in this ballpark that are also good at conversation? They always show off their software capabilities, but can it talk naturally? make creative stories?
The closest to this that's also good at prose is gemma4 26b a4b. That's what I'm using for a fully-local "Tunnel Goons" (light D&D-style game) harness where multiple player agents and a DM agent improvise a story together.
I haven't tried inkling so I have no idea how it speaks, writes, and tells stories. I'd love to know what other small models folks are having success with in role-play, gaming, creative writing environments.
Big fan of open weights but those Terminal Bench numbers are pretty abysmal, and I find that tends to track "agent usefulness" more than most SWE benchmarks.
The demo video was 32 tok/s and did a 15X speedup when generating the code? Weird that they didn't publish token speeds for this laptop setup they recorded. I'm worried this tri-loop setup is going to be too slow. QFN is impressive but lately been feeling a little brain-fried by its prose.
Quite an interesting model, but the second of my test prompts in "extra high" thinking in Unsloth really underscore what they say about knowledge here:
> While looping improves the model’s reasoning, it doesn’t enable its weights to store more facts—knowledge is mostly a function of total parameter count. Inkling 2 Mini’s score on AA-Omniscience, a measure of factuality, is in line with other models of the same size [...] but knowledge-intensive benchmarks doesn’t improve with more passes...
> Inkling 2 Mini overcomes the knowledge gap by efficiently using search and tools.
I tested the Unsloth Q4 version.
My prompt asked it to write a singleton class, single file plugin for WordPress that hooks user login and stores last login time as metadata. It has some code style requirements. The prompt also asked it to make no tool calls so it didn't search and only used its knowledge.
The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish. Its thinking suggests it was going to make at least one mistake.
Both Qwen 3.6 35B and Gemma 4 26B can cope with this task, from memory, in far fewer tokens (even taking into account Qwen's tendency to loop in thinking).
Rerunning the same prompt with tool use allowed, it searched with browser use, pretty effectively, and got to a solution, but it still made the mistake the thinking traces suggested it would earlier (wrong hook name). Other than that it wrote good code.
But 20K-ish tokens in, I could then not get it to search to make the required correction — it would repeatedly EOT rather than actually run the tools it was saying it would call. Whether this is the model failing or Unsloth's harness and the model disagreeing, I don't know. Quitting and reloading the model to continue the conversation didn't help.
It worked through my usual MySQL-5.0-compatible max score per category prompt nicely, though it didn't volunteer the suggestion of adding an index for speed, but it doesn't outperform LFM 2.5-24B-A2B, which is a non-looping model of similar size. It does work through SQL in its thinking traces quite nicely, though.
Car wash problem got a correct answer quite quickly, albeit with a surreal caveat (why do they always seem to need to offer a caveat, no matter how weird)
So far, I think I would run Muse Glimmer or Nemotron Bolt over this; Muse Glimmer would output tokens more slowly but still get to a result much more quickly.
hate these dark practices. they say click here to try it on our site. then u click. auth wall. fine you give it all your info. sunk cost is maximized. then they hit you with the credit card wall.
Somewhat astonished that thinking machines would do this considering that looping transformers a.k.a. neuralese is safety wise way harder to keep track of what the LLM is actually doing. by the way, thinking machines CEO is one of the people that quit openAi over safety concerns.
Seems like this is the successor to Qwen 3.6 35B that we've been waiting for? I'm not aware of any other recent models of this size that can run well on mid-tier GPUs without severe quantization.
mattnewton | 3 hours ago
From a research perspective, I'm even more interested in the varlooped variant they're releasing https://thinkingmachines.ai/model-card/tml-d24-varloop-resea... I'm not aware of any other working open source models like that.
kcb | 3 hours ago
liuliu | 3 hours ago
EDIT: by "open problems", I meant the problems I don't really formulate how to solve it first in my mind and then throw to it. I just describe what I want and let it to figure it out.
walrus01 | 3 hours ago
https://huggingface.co/thinkingmachines/Inkling-2-Mini
https://huggingface.co/unsloth/Inkling-2-Mini-GGUF
In terms of the active parameters and token/s on slow or less capable hardware it seems like it should be in the same general performance category as qwen 3.6 35B A3B.
jakswa | 3 hours ago
jakswa | 3 hours ago
hrncn | 3 hours ago
neya | 3 hours ago
christkv | 3 hours ago
topaztee | 3 hours ago
christkv | 2 hours ago
joerambo808 | 25 minutes ago
rapind | 3 hours ago
christkv | 2 hours ago
rsolva | an hour ago
Iolaum | 3 hours ago
P.S. Qwen3.8-Flash-Next is also a very good model for Srix-Halo/128Gb Framework Desktop but I liked to have more RAM for other things as well even when I was running it in Q4. Usable though.
montroser | 3 hours ago
Hope that folks find this model more amenable to local usage - the sizing was largely informed by what folks on this subreddit ask for the most haha!
There are two particularly interesting aspects of this model for local users:
We do our training end-to-end in nvfp4, and although we haven't exhaustively benchmarked this, folks like Unsloth have noticed that our models seem to respond very well to being quantized, maintaining 90% top-1 even at at 1-bit quantization.
This model is looped, which allows more depth at the same parameter count, which we particularly notice seems helpful for coding tasks. :)
This is our first model targeted at local users, so let us know how this model is and anything you want to see out of future iterations!
https://www.reddit.com/r/LocalLLaMA/comments/1y1diuk/thinkin...
LPisGood | an hour ago
cmrdporcupine | an hour ago
Not sure the impact on kvcache though.
teravor | 39 minutes ago
then there was a fully looped model (forget the name) where all the layers were looped (which is an obvious mistake as it's known LLM's have token processing and reasoning stages) and showed some decent results.
and now the concept worked itself into the mainstream.
calebkaiser | 28 minutes ago
If you're curious, it's a very cool paper! https://arxiv.org/html/1807.03819v1
dofm | 48 minutes ago
kroaton | 44 minutes ago
calebkaiser | 32 minutes ago
That's just a general simplification though. One of the key motivators in this general area of research is adaptive latent reasoning, meaning in theory, some tasks only require 2 loops and some require 8.
There's a lot of different approaches here and many of them approach the problem from different traditions. Some looped transformers use a fixed loop count, others follow after deep equilibrium models and iterate towards a fixed point or some other stopping mechanism.
Some fun reading if you're interested in this field:
- TinyRecursiveModels - Blew up a couple of years ago when it performed really well on ARC AGI 1 using a comparatively small model with a large task-level table of embedding prefixes. https://github.com/samsungsailmontreal/tinyrecursivemodels
- Sebastian Raschka's piece on the general architecture patterns we see today: https://sebastianraschka.com/llm-architecture-gallery/looped...
stavros | an hour ago
To me, this means they probably won't.
blurbleblurble | 45 minutes ago
dofm | 35 minutes ago
And then I test it in comparison and I remember Goodhart's Law.
This model, for sure, is interesting. But there's more than one way to peel a potato, and again and again I find that in practice these benchmarks don't really seem to mirror normal "coding assistant" uses.
People have got really hooked on benchmarks and token speeds, but a fast-outputting model that wastes endless tokens on thinking and then gets it wrong is not much use.
And no amount of tool-calling search really seems to "close the knowledge gap" when a small model is absolutely insisting something exists when it does not.
jakswa | 3 hours ago
jakswa | 3 hours ago
summarity | 3 hours ago
holistio | 3 hours ago
efficient_dairy | 33 minutes ago
gchadwick | 3 hours ago
I think this may be the first public example of a looped transformer model good enough for real world use cases?
rsolva | 3 hours ago
AndrewKemendo | 3 hours ago
Also do you know what the best local setup is?
rsolva | 2 hours ago
jakswa | 2 hours ago
edit: found some oudated mesa/radv/shaderc/glslang packages to update, and up to 34 tok/s now. MORE MORE
edit2: trying to see if this dspark drafter is usable and will give a good speed boost, didn't load up on first attempt. I'm probably too early [5-10min later] ... found an unsloth llama.cpp build with vulkan dspark support and it actually slowed down decode :(
rsolva | 2 hours ago
jakswa | 2 hours ago
enlyth | 2 hours ago
jakswa | 2 hours ago
shadowpho | an hour ago
IAmGraydon | 2 hours ago
OK, but please come back and let us know your results!
rsolva | 2 hours ago
Flash Attention (FA) is not supported on the current llama.cpp (at least not for ROCm) for this model so KV cache is forced to f16 for the value (V) and I cannot lower it to q4 go get decent KV cache. But the key (K) could be set to q4, and that got me about 100k of ctx. Current benching shows ~28 tok/s decode and ~481 tok/s prefill.
It's a start, but I expected better speeds even on a non ideal setup. I'll keep experimenting!
xlayn | 2 hours ago
I'm seing around 2800t/s on pp at the very beginning of the conversation, and then going to around 800 by the 256k context size...
I've also update the value in the gguf to have instead of context size 256k be triple and have use it all the way to around 600k context size
use the cache eviction branch https://github.com/alainnothere/llama.cpp/tree/disk-cache-ev...
llama-server -m /path_to_model/Qwen3.8-27B-UD-IQ4XS_u.gguf -np 2 --host 0.0.0.0 --port 8089 --kv-unified --temp 0.2 --top-k 20 --min-p 0.0 --cache-disk-path /path_to_store_conversations_to_disk --cache-ram 100 --cpu-range 0-23 --reasoning-budget 4096 --reasoning-budget-message "... I am thinking for too long -- let me gather more info about the task." --no-cache-idle-slots --spec-draft-auto --spec-draft-sampling probabilistic --spec-draft-n-min 1 --spec-draft-p-min 0 --spec-ngram-auto --spec-ngram-mod-size 35544320 --spec-ngram-mod-cache /path_to_storage_of_ngram_mod_this_is_specific_per_model/ngram-mod-qwen3.8-27b.bin --fit on --device Vulkan3,Vulkan2,Vulkan1 --fit-target 1024,2560,2560 --ctx-checkpoints 25 --checkpoint-spill-max 25 --cache-type-k f16 --cache-type-v f16 -ctkd q8_0 -ctvd q8_0 -devd Vulkan3 --spec-type draft-mtp,ngram-mod --spec-draft-n-max 7 --spec-ngram-mod-n-min 7 --spec-ngram-draft-n-max 31 --spec-rs-depth 24 --fit-verify on
Use this if you have 3 cards, if you have less maybe just remove it -b 8192
In case you want to know how many tokens you have use, --token-ledger /path_to_where_you_want_to_store_stats/token_ledger.json
here is how logs look like I saw a 350t/s on tg but was because you hit ngram mod tables over things the model has already seen
457.04.985.808 I srv print_line: token ledger: requests = 27077, prompt = 3928887769 (cached = 3874707808, processed = 47151296), generated = 22126308 (reasoning = 15129037), t> 457.04.985.913 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77431, ttft 1580.0 ms (http 20.4, tokenize 65.1, queue 247.1, select 0.0, restore 2.7, prompt 1244.3),> 457.05.202.640 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user) 457.05.202.757 I slot get_availabl: id 0 | task -1 | selected slot by LCP similarity, f_sim_best = 0.994 (> 0.100 thold), f_keep = 1.000 457.05.206.552 I slot launch_slot_: id 0 | task 77458 | processing task, is_child = 0 457.05.207.188 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81032..81036 of 81552 457.05.246.405 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81036..81548 of 81552 457.05.310.149 I srv lazify_check: checkpoint: converted pos_min=80943 to disk-backed ('cp_64d89c6d7ff76923_80943.bin', 149.6 MiB released) 457.05.872.722 I slot operator(): id 0 | task 77458 | prompt processing batch, tokens 81548..81552 of 81552 457.05.943.921 I srv lazify_check: checkpoint: converted pos_min=81035 to disk-backed ('cp_64d89c6d7ff76923_81035.bin', 149.6 MiB released) 457.09.000.990 I slot print_timing: id 0 | task 77458 | n_gen = 340, tg = 112.88 t/s, tg_3s = 113.20 t/s 457.11.161.223 I slot print_timing: id 0 | task 77458 | prompt eval time = 790.96 ms / 520 tokens ( 1.52 ms per token, 657.43 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | eval time = 5163.27 ms / 642 tokens ( 8.06 ms per token, 124.15 tokens per second) 457.11.161.226 I slot print_timing: id 0 | task 77458 | total time = 5954.23 ms / 1162 tokens 457.11.161.227 I slot print_timing: id 0 | task 77458 | graphs reused = 72319 (pp skipped 525) 457.11.161.228 I slot print_timing: id 0 | task 77458 | graphs prebuilt = 7295 (hit 6975, miss 320) 457.11.161.229 I slot print_timing: id 0 | task 77458 | early prebuilt = 28686 (fit 44246, hit 26411, miss 2275, aborted 0, 96584.2 ms host) 457.11.161.238 I slot print_timing: id 0 | task 77458 | draft acceptance = 0.83204 ( 535 accepted / 643 generated), mean len = 6.00 457.11.161.243 I slot print_timing: id 0 | task 77458 | draft auto = n=6 | 1:55.4 t/s [2.00 tok / 36.1 ms] (4) 2:67.7 t/s [2.50 tok / 37.0 ms] (4) 3:101.0 t/s [4.00 tok / 39.6 ms> 457.11.161.251 I slot print_timing: id 0 | task 77458 | draft by impl: draft-mtp | steps 101 | built 5.5 | offered 5.5 | accepted 4.9 (0.90) | full 80 | step 45.2 ms | 131.5 t/s > 457.11.161.257 I slot print_timing: id 0 | task 77458 | draft by impl: ngram-mod | steps 6 | built 64.0 | offered 14.3 | accepted 6.0 (0.42) | full 1 | step 90.8 ms | 77.1 t/s | > 457.11.161.258 I slot print_timing: id 0 | task 77458 | ngram policy: auto on | cap now 12 | cap min 7 median 12 max 31 over 6 chains | cap bound 6 | whole ema 0.17 | resume 0 | > 457.11.161.259 I slot print_timing: id 0 | task 77458 | spec rollback: prebatch restores 1 | host checkpoints 0 | checkpoint restores 0 457.11.161.260 I slot print_timing: id 0 | task 77458 | chain truth: ngram | chains 6 | censored 0 | would land mean 7.0 | full 0 | hist 0:0 1-4:1 5-8:3 9-16:2 17-32:0 33-48:0 49> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ngram gate: held 38 | agree 5 (would land 0:0 1-4:1 5-16:4 17+:0, pending 0) | disagree 33 (would land 0:33 1-4:0 5-16:0 1> 457.11.161.261 I slot print_timing: id 0 | task 77458 | ext shadow: mtp+ngram | attempts 101 | hits 85 | tested 68 | landed mean 0.4 (over tested) | hist 0:65 1-4:1 5-8:0 9-16:2 > 457.11.161.262 I slot print_timing: id 0 | task 77458 | alt table: n=48 | attempts 101 | hits 45 | would land mean 0.0 | hist 0:45 1-4:0 5-8:0 9-16:0 17-32:0 33-48:0 49-64:0 65+:> 457.11.161.263 I slot print_timing: id 0 | task 77458 | mtp ahead: tried 53 (0 after a p_min stop) | verify full 49 | bonus matched 48 (0 after p_min) | too short 0 | served 48 s> 457.11.161.265 I slot print_timing: id 0 | task 77458 | mtp confidence: stop cap:101(acc 4.9) pmin:0(acc 0.0) other:0 | p bins landed/tested: <0.5 1/3 0.5 0/1 0.6 1/1 0.7 0/0 0.8> 457.11.163.234 I slot release: id 0 | task 77458 | stop processing: n_tokens = 82195, truncated = 0 457.11.163.426 I srv print_line: token ledger: requests = 27078, prompt = 3928969321 (cached = 3874788840, processed = 47151816), generated = 22126950 (reasoning = 15129071), t> 457.11.163.526 I slot rq_write_fin: id 0 | task -1 | request timeline: task 77458, ttft 884.3 ms (http 21.2, tokenize 68.3, queue 0.1, select 0.0, restore 3.3, prompt 791.0), gen> 457.11.396.231 I srv handle_compl: task created: conversation_id=64d89c6d7ff76923 (tokens 0..18489, system+first_user)
yread | 3 hours ago
https://www.sensetime.com/en/news/sensenova-loop-9b-release-...
lwarfield | 3 hours ago
https://dnhkng.github.io/posts/rys/
mapontosevenths | 2 hours ago
That took me from needing to rent space in avast to just running it locally on my sparks and getting comparable results.
I also discovered that you can fix a lot of the damage caused by quantization the same way. Just do 200-500 rounds of parent student distillation with everything frozen but the residual, often the quantized version will improve drastically.
AndrewKemendo | 3 hours ago
2001zhaozhao | 3 hours ago
Dethroning Qwen 3.8 is a big deal too, and I like their section about CoT monitorability. (This model beats Qwen on benchmarks - TML is not known to benchmaxx nearly as hard as Qwen.) I guess TML really are a bunch of rationalists who know what they're doing, sort of like Anthropic.
bpavuk | 2 hours ago
yes
> sort of like Anthropic.
no. Anthropic are not "rationalists," they are absolute fear-mongers. from "no software developers in 6 months" to "this could enable bioterrorism so we do not even let you mention the word 'virus' in any context", I do not see a hint of rationality.
ungovernableCat | 2 hours ago
Maybe they'll chill out a bit?
thereader12 | 2 hours ago
xvector | 19 minutes ago
lenerdenator | 2 hours ago
I honestly would rather have nice, focused-purpose local models than huge "let's make AGI" datacenter-hosted models. Will have to try it tonight when I get home.
segmondy | an hour ago
gok | 2 hours ago
Idea being you can configure at run time how much thinking you let the model do per token.
xlayn | 2 hours ago
monksy | 2 hours ago
WiSaGaN | 2 hours ago
stillpointlab | 2 hours ago
halJordan | 2 hours ago
KeplerBoy | 2 hours ago
bigyabai | 2 hours ago
kypro | 2 hours ago
Presumably v. good for offensive use cases too.
And a looped model?
This and GRH is +0.1% on my p-doom today.
The trends of capabilities and efficiency we've seen over the last few weeks should terrify anyone with any sense.
This will not end well but we'll have some nice AI generated dashboards until then I guess.
bigyabai | 2 hours ago
But hey, you're welcome to increment your P(doom) until the year 2057 if it makes you feel better. Maybe the government will finally put their foot down with the release of Qwen 28, which according to OpenAnthropicTimeWarnerPepsico will have unprecedented puppy-kicking use cases that necessitate regulation.
hypfer | 2 hours ago
bigyabai | an hour ago
lilytweed | 2 hours ago
I'm willing to wager that most open models will go looped in the wake of this.
Wonder what it means for safety and interpretability at this level of availability.
coder543 | 2 hours ago
https://huggingface.co/Nanbeige/Nanbeige4.2-3B
https://huggingface.co/nvidia/NVIDIA-Nemotron-Labs-Dory1.0-4...
https://huggingface.co/sensenova/SenseNova-Loop-9B
But, Inkling 2 Mini is the first open weight MoE looped transformer of any note that I've seen (the others are dense models), and the benchmark scores really do look incredible.
If it can outsmart Qwen3.8-27B while using substantially fewer active parameters... this is going to be really great.
twothreeone | 58 minutes ago
And the fact that they trained it as a looped model in 4-bit native is also kind of impressive to me as well, because it combines recurrence, routing, and narrow-format - all of which adversely impact numeric stability during training.
XenophileJKO | 2 hours ago
Which is counter to the way most open weight models have been leaning. Many current models are MoE (Mixture of Experts) which by their very nature are parameter heavy. They are trading parameter efficiency for compute efficiency which is kind of the wrong tradeoff for current consumer hardware like gaming GPUs.
coder543 | 2 hours ago
Consumer hardware has limited memory, limited memory bandwidth, limited compute, and limited storage. Any model designed for consumer hardware needs to tackle all four, while still optimizing for intelligence.
MoEs save on both memory bandwidth and compute compared to a same-size dense model, and importantly, MoEs can still run quickly even with partial offloading. People have even demonstrated getting pretty good performance out of MoEs streaming from SSD with careful optimization.[0] Dense models fall apart if they aren't run entirely in the fastest memory pool.
Memory bandwidth is scarce on consumer laptops and smartphones. Looping demands even more memory bandwidth than running the same parameters once, so it is not an obvious win. If looping can make the model significantly more intelligent, it might be worthwhile. It is interesting to see a MoE combined with looping, as with Inkling 2 Mini. On your point about memory being the constraint, looping increases the size of Inkling 2 Mini's KV cache because the looped layers don't share KV between loops.
With DSpark-style specdec getting better and better, dense models make more sense than they used to since you can amortize the memory bandwidth better. The problem is then prefill (prompt processing), since specdec does nothing to help with that, and processing long inputs (search results, reading documents, etc) will still feel very slow. A MoE can breeze through that same prefill since it doesn't need as much compute.
With the introduction of Engrams/PLEs, model designers now have another axis for adding knowledge to models at the cost of storage, with comparatively little additional compute or memory bandwidth.
There are tradeoffs on everything. Despite my own extensive testing of different models on different hardware setups, I have not found any one-size-fits-all model architecture for consumer hardware. It is a hard problem. Small models are getting better all the time, and I think even smartphone-sized models will pass the threshold of usable intelligence for most people in the next year. Spark-X2.6-4B[1] and LFM3-1.7B[2] are surprisingly good for their sizes. Inkling 2 Mini looks great, supposedly offering Qwen3.8-27B's intelligence with much less compute required.
[0]: https://huggingface.co/openbmb/MiniCPM-V-4.7-35B-A3B#on-devi...
[1]: https://huggingface.co/XHToken/Spark-X2.6-4B
[2]: https://huggingface.co/LiquidAI/LFM3-1.7B
Eisenstein | an hour ago
Important caveat: if not fully offloading[0] then the MoE model's prefill speed completely tanks, as prefill is heavily dependent FLOPS and parallelization and CPUs suck at that compared to GPUs.[1]
[0] 'Offload' is being used in the llamacpp sense of using the GPU instead of CPU
[1] Generally, prefill is bottlenecked by compute while generation is bottlenecked by memory bandwidth
coder543 | an hour ago
When you're prefilling, you can afford to amortize the cost of sending the experts to a scratch space on the GPU then running a bunch of tokens through that set of experts, then repeating with the next set of experts, and so on. There's no reason to compute those on the CPU, and llama.cpp already does move prefill to the GPU by default for partially-offloaded MoEs.
Where performance would still tank is if you don't ensure the microbatch (ubatch) is large enough to amortize that cost, and if you're prefilling only a small input, it's not going to reach full speed. For a large input, you can think of each expert as visiting the GPU once for every ubatch of input tokens. A larger ubatch means fewer transfers.
Try increasing `-ub` (2048 is a sane default, but 4096 or higher can sometimes be desirable), and watch the partially-offloaded MoE prefill performance magically go way up on llama.cpp for large inputs.
EDIT: Also, if you haven't tried replacing `--n-cpu-moe` with `--cpu-moe` + `--moe-cache-mib` (with all of your GPU memory that you can spare), it is even better than `--n-cpu-moe`, since real world models don't distribute work over the experts in a perfectly random way. The moe-cache speeds up decode/output tokens per second.
cubefox | an hour ago
We already know from official model cards that the GPT-6 models are substantially less monitorable than other (non-looped) models. Which is not surprising, since they are doing some amount of Neuralese thinking between each token.
Almondsetat | 2 hours ago
hunterloftis | an hour ago
I haven't tried inkling so I have no idea how it speaks, writes, and tells stories. I'd love to know what other small models folks are having success with in role-play, gaming, creative writing environments.
Almondsetat | an hour ago
causal | 2 hours ago
armcat | 2 hours ago
dofm | an hour ago
sourcecodeplz | an hour ago
jakswa | an hour ago
AtHeartEngineer | an hour ago
dofm | an hour ago
> While looping improves the model’s reasoning, it doesn’t enable its weights to store more facts—knowledge is mostly a function of total parameter count. Inkling 2 Mini’s score on AA-Omniscience, a measure of factuality, is in line with other models of the same size [...] but knowledge-intensive benchmarks doesn’t improve with more passes...
> Inkling 2 Mini overcomes the knowledge gap by efficiently using search and tools.
I tested the Unsloth Q4 version.
My prompt asked it to write a singleton class, single file plugin for WordPress that hooks user login and stores last login time as metadata. It has some code style requirements. The prompt also asked it to make no tool calls so it didn't search and only used its knowledge.
The model burned through 19200 thinking tokens on this task (which was the hard limit I had set) and didn't finish. Its thinking suggests it was going to make at least one mistake.
Both Qwen 3.6 35B and Gemma 4 26B can cope with this task, from memory, in far fewer tokens (even taking into account Qwen's tendency to loop in thinking).
Rerunning the same prompt with tool use allowed, it searched with browser use, pretty effectively, and got to a solution, but it still made the mistake the thinking traces suggested it would earlier (wrong hook name). Other than that it wrote good code.
But 20K-ish tokens in, I could then not get it to search to make the required correction — it would repeatedly EOT rather than actually run the tools it was saying it would call. Whether this is the model failing or Unsloth's harness and the model disagreeing, I don't know. Quitting and reloading the model to continue the conversation didn't help.
It worked through my usual MySQL-5.0-compatible max score per category prompt nicely, though it didn't volunteer the suggestion of adding an index for speed, but it doesn't outperform LFM 2.5-24B-A2B, which is a non-looping model of similar size. It does work through SQL in its thinking traces quite nicely, though.
Car wash problem got a correct answer quite quickly, albeit with a surreal caveat (why do they always seem to need to offer a caveat, no matter how weird)
So far, I think I would run Muse Glimmer or Nemotron Bolt over this; Muse Glimmer would output tokens more slowly but still get to a result much more quickly.
gregsadetsky | an hour ago
recipe: https://gist.github.com/gregsadetsky/85e8b8ea91430dc826611b9...
jedisct1 | an hour ago
purplecats | an hour ago
mentalgear | an hour ago
nunodonato | an hour ago
zergrush | an hour ago
nitroedge | 27 minutes ago
hatthew | 27 minutes ago