The efficient frontier of LLM inference

76 points by philipkiely 5 hours ago on hackernews | 19 comments

this is a nice and concise writeup. what's striking to me is that these techniques really have not changed in /years/. sure, precision has become slightly lower, spec decoding acceptance has gotten slightly better and the complexity of parallelism is trickier with mixture of experts. but no new concepts in a very long time!

the absolute most impactful improvements for inference comes at architecture design time. I firmly believe everyone who cares about impacting model efficiency should look there

[OP] philipkiely | 4 hours ago

I think the biggest net new recent technique is P/D disaggregation. And that spec dec is very different now especially post DSpark/DFlash.

But overall yes the fundamentals of LLM performance optimization have been remarkably stable over the last few years.

: brrrrrm | 3 hours ago
perhaps its unfair to say this in hindsight, but it's a fairly straightforward application of little's law that's been around for some time
https://arxiv.org/html/2401.09670v2

datadrivenangel | 4 hours ago

The author does not deeply mention that quality/intelligence is a third dimension here in addition to throughput and latency, and the frontier is jagged so quality and intelligence require bespoke benchmarks to evaluate tradeoffs for speed and cost.

: [OP] philipkiely | 4 hours ago
These are both good points that I attempted to cover, quotes:
> In practice, the efficient frontier is very jagged. Rather than a smooth, continuous line between outcomes, small changes can have big impacts. These cutoff points are often unintuitive and must be discovered empirically through sweeps.
> However, quantization introduces a new set of tradeoffs between quality and serving efficiency. This is a particularly jagged frontier, where a large degree of improvement to serving efficiency is possible with little-to-no reduction in model quality, especially when using microscaling floating-point number formats like MXFP4 and NVFP4.
Would appreciate ideas on how to explain in greater depth

calclavia | 4 hours ago

good recap on the recent inference techniques!

ttoinou | 4 hours ago

   Inference techniques either move a deployment along the latency–throughput frontier or push the entire frontier out, creating more efficiency to allocate.

This is a tautology. You can say that with anything. Gastronomy techniques will make a previous recipe better, or create a new recipe better than others, or a mix of both.

Ifkaluva | 2 hours ago

The point is to classify them into two kinds. The kind that shifts the frontier is more powerful, since improves capabilities without incurring tradeoffs.

: [OP] philipkiely | 2 hours ago
I also wrote this as somewhat of a defense of the techniques that don't move the frontier -- there is a lot of value in being able to pick a point on the curve.

jumploops | an hour ago

> Speculative decoding is the process of guessing which tokens a model might generate, then validating those guesses.

As a computer engineer, it’s always interesting to see optimizations applied at different levels of the stack.

Speculative execution became pretty popular in the 90s, eventually used in basically every x86 design.

Then in the mid-2000s the Speculator[0] paper brought that concept to distributed systems, which we’re still seeing work on[1][2].

Everything old is new again (:

[0]https://www.cs.princeton.edu/courses/archive/fall07/cos518/p...

[1]https://www.usenix.org/system/files/osdi25-shen-weihai.pdf

[2] https://www.microsoft.com/en-us/research/publication/distrib...

freakynit | an hour ago

Can we expect similar issues such as spectre and meltdown that intel experienced with speculative execution.. but, in the form of prompt injection/poisoning?

: rf15 | 39 minutes ago
Ok, I'll bite: no, considering these are very different domains and you don't get system access by getting the wrong speculative branch for your next text token, you just get a slightly different (but probably still related enough) text.

mathisfun123 | 53 minutes ago

> optimizations applied at different levels of the stack

That's because it's just "guess and check" not some deep universal insight.

arjie | an hour ago

You know what I'm curious about? Whether you have brand guidelines inside the company, a Claude skillset, or the blog post author makes the charts in line with the brand colours and so on.

[OP] philipkiely | 28 minutes ago

I draw my diagrams on notecards and send them to our designer who brings them to life.

The images start out looking like this: https://philipkiely.com/images/blogs/how-to-write-a-book/des...

: arjie | 16 minutes ago
Makes perfect sense. The classic way!

fsckboy | 58 minutes ago

"the efficient frontier" is an important landmark of (investment) portfolio theory. It proves/explains/illustrates how you can combine selections from a diffuse cloud of individual investments and still land on a frontier that is better than any of your individual choices. It's the entire basis of "diversify your portfolio".

The efficient frontier of LLM inference is a line, not a frontier.

this is a frontier:

https://upload.wikimedia.org/wikipedia/commons/e/e1/Markowit...

no matter how good is something a smart person writes down, a pleb will come along and try to hang on its coattails.

If you want to steal an idea for this, steal indifference curves, they'd make more sense.

censor25 | 24 minutes ago

Nice read. I was wondering what can one do to get into inference engineering as simple theoretical knowledge is not sufficient and switching profiles is tough for someone with years of experience.

qingcharles | 21 minutes ago

> A model is a “frontier model” if it offers the highest degree of intelligence at a given cost or size.

I would define a "frontier model" as offering the highest degree of intelligence at any cost, or without regard to cost. The frontier today is clearly Fable/Mythos, with the "efficient frontier" at Opus/Sol.