Another day, another model and another pelican :-)
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.
> better models consistently renders pelican in higher quality
The article literally avoid making this argument and gives counterexamples to this statement.
As mentioned elsewhere, the benchmark introduces bad pelicans in the training set. What I'm curious about however, if it's possible for a human artist to "poison" the benchmark by releasing some really good pelicans svgs and have all future models output their version.
> How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.
My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one.
Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.
> they're comparing to similar capability llm models, not humans
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
Well first of all, any non-trivial use of LLMs is going to be orders of magnitude more tokens than this, usually multiple millions at minimum. Benchmarks are just benchmarks after all.
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...
It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website
the nature of the test was to see if the models can effectively compose an image of a novel concept outside the training set. If they are trained on it, it ceases to be an interesting test to some extent.
it's still interesting because there's no pelican-on-bike model, and if you're training a model well enough, then it should be obvious when a model has reached "AGI" or whatever.
I would urge you to re-read the blog post you are commenting on. It pretty clearly explains how it is an interesting test independently of "see[ing] if the models can effectively compose an image of a novel concept outside the training set".
> Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?
I would think the opposite because unless people have been hand drawing these with high quality, the training would be on much crappier versions that old AIs have done.
They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.
More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.
Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.
From the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.
This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.
We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.
More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”
Maybe it gets posted every time because besides a personal believe by the person popularising this "benchmark", there is no reason to assume that certain labs aren't intentionally training to game this and every other lab at least unintentionally gets improvements for this specific combination of animal and action because the internet is full of both good and bad examples, often ranked, which does inevitably become training data.
I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
> I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts
Please share those again!
One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).
Evidence in the other direction (that they're able to generalize) is that I can't think of any LLM currently that can't create usable (placeholder) SVG icons, I tried a bit before the pelican became popular and it was abysmal.
Happy to, here one example where Grok 4 Fast, despite producing a fairly consistent pelican [0], did severely worse in a similarly outlandish scenario along with Haiku 4.5 and GPT-5 for context: https://news.ycombinator.com/item?id=45599403
> [...] a really great pelican riding a bicycle and then a terrible sloth riding a skateboard [...]
Happy to play ball. You made a blog post a few weeks back on one of the Qwen models with the eye-catching title "Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7" [1].
Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5
Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. Whether this is a case of training data being unsanitized or intentional benchmark targeted training, I cannot say, but it is the case.
A massive delta in favour of Opus 4.7, despite the pelican Qwen3.6-35B-A3B produced being noticeably better as you rightly pointed out. What does that tell us? Whether intentional or not (with such deltas, I do have my suspicions), any eval with such a delta is clearly polluted and can not be a source of information, especially as its continued existence does hinge on you testing similar prompts in private as a sanity check, yet by your own admission never noticing the plainly apparent delta in quality. I specifically stuck with the skateboarding sloth too, to keep it as fair as possible and found this in less than 5 minutes...
I would not critique your use of this fun benchmark the way I tend to if I did not have evidence to back up my position, including private evals beyond SVGs that I can reliably use to point out major deviations between what a models claimed performance is according to major benchmarks vs the actual performance outside these known test cases.
I will also say that while I have a lot to be critical of regarding Anthropics modus operandi, especially how they present interesting findings like their j-space work, which I found was irresponsibly anthropomorphic in their reporting, especially as this wasn't a first in model interpretability, but mainly a leap due to being applied to a larger model, but of all the labs, they are the ones that never underperform my evals vs public ones and they appear to strictly keep their training data sanitised.
Happy to discuss public vs private evals and the merit of each if you'd like, I do appreciate your reporting in general but just think the SVG benches have become evidently polluted, which is also why even simple queries in my benchmarks are private. Just saw Thinking Machines Inkling model succeed in certain queries that neither Fable 5, nor GPT-5.6 Sol on any reasoning level managed, which I feel is valuable to truly gauge where we are at. Informs my work with models, my views of the industry and my assessment of the future these tools have, along with how to best implement them to enable better UX.
Respectfully, did you? The comment was specific to doubting the believe simonw has that labs are not training [0] specifically for this task, which is exactly what simonw wrote in the post [1], that it is a believe of his that they don't. He did not mention any kind of evidence or any piece of information that would indicate that the commenter didn't read the blog post.
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
I'll note there's a difference between "pelicans on bikes aren't part of the training set" and "I’m still not convinced that labs are training for the benchmark".
I'm sure all sorts of crap pelican riding bicycle SVGs have ended up in the huge crawls of data that the labs feed into their pre-training steps.
What I'm questioning here is that there are labs who have sat down and deliberately tested and tweaked the performance for this particular task, independent of general model improvements.
The one exception here is Gemini, who have clearly invested a lot of effort in SVG tasks. I have no idea if my stupid benchmark influenced that decision!
Gemini have boasted about how good they are at pelicans riding bicycles, frogs on penny-farthings, giraffes driving a tiny car, ostriches on roller skates, turtles kickflipping skateboards, and dachshunds driving a stretch limousine. So if they trained for the test they did at least expand it a whole bunch! https://twitter.com/JeffDean/status/2024525132266688757
> What I'm questioning here is that there are labs who have sat down and deliberately tested and tweaked the performance for this particular task, independent of general model improvements.
We are going from pretty good pelican to jumbled mess with a similarly silly, but different prompt across multiple models from multiple labs, both Western and Eastern, both Open Weight and Closed.
Yes that's the obvious thing to do and why straightforward variants of known tests would also be treated as contaminated by anyone being even somewhat rigorous.
I don't know why the standard is is to be sure that it is happening versus it being a plausible risk of making the results useless.
The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!
Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.
In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.
Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
You know what, that's actually something I hadn't considered before. There's definitely a bias towards a pelican cycling from left to right on a red bicycle against a blue sky and green grass.
Blue sky and green grass aren't that surprising, but the color and direction are interesting.
When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.
There was a glorious moment when I thought that the Chinese models were more likely to produce right-to-left cycling pelicans, but sadly that trend didn't seem to hold up.
That was my first thought too, I wonder if it works the same in countries speaking arabic (as that's the first one i could think of that's a language with truly no-buts right to left writing).
Yes, people will usually post or draw a bicycle right to left which is going to ve opposite of what normally is drawn. I tried the prompt in arabic for many models and I don't recall any adjusting it based on that difference at least culturally speaking.
It is. All over the Arab world, imagery in ads is “backwards” and I believe several companies will flip their ads horizontally, and UI localization involves flipping graphics.
Do the gears and stuff sit on the other side of the bike in the Arab world? Otherwise I'd expect cycling ads to still show a bike going from left to the right, considering https://news.ycombinator.com/item?id=48951828
The other thing to consider (as someone who frequently take a photos of their bike) the common direction has the drive side out! In cycling forums it is sacrilegious to post a photo of your bicycle without showing the drive side.
Beat me to it - but I had the same thought. Most amateur and nearly all professional studio photographs of a bicycle will have it drive side out so I expect this plays some role in it.
Took some searching and sleuthing to actually figure out what "drive side out" means, as I'm just a casual "from A to B" cyclist: apparently this is referring to the side the chainset, chainwheels and all those things are on.
I haven't seen many AI works that produces a pelican on a bicycle done in a "Ligne Claire" style, for example.
I guess AI's narrows down the output probability space drastically and converge on some agreed upon aesthetics. Works great for computer programs but bad for art.
I have done some variation of the other animals, also for something more tricky where they need to calculate things, I ask them to draw an SVG at a certain angle.
For example: "generate an SVG of a chessboard seen from a 45 degree angle slightly higher POV" or "generate an SVG of a basketball court from a TV broadcast perspective".
Bicycle color, grass color and sky color are all part of the prompt.
>Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground
That wasn't the prompt. That text was generated by asking the model to describe an image and feeding it a rendering of the SVG it had previously generated.
There's a bias in the direction all things face. You can ask these models to generate a thing animal, car etc and you will notice that 90% of them will converge towards the same sort of results. If you ask for something rotating, 90% of them will rotate right and a few odd ones will rotate left.
What's interesting is that given the fairly general and short in length prompt for the test, none of the models are attempting things like more discrete details of the bike. Such as showing V-brakes or dual 160mm disc rotors, rear derailleur, water bottle in a bottle cage, panniers, lights, saddlebag, the rider wearing a helmet, or other details that might be found on as vague a description as "a bicycle".
The models are already brilliant at that. My own todo app generates 128x128 pixel art icons for my todo items. They are mind blowingly creative and funny.
> the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model
You would not expect that to happen if the models trained on the unrecognizable mess, right?
> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts
And the labs clearly did focus on improving image rendering.
> they have a uniform style
SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.
I’m not suggesting Simon’s pelicans in the dataset are having a meaningful impact. I’m expecting that a company like ScaleAI has a product along the lines of “benchmax dataset: SimonW’s Pelican on Bikes test” which is a private curated series of well-drawn SVGs of animals riding vehicles for training and RL.
If they're benchmaxed on SVG pelicans then the outcome of that has still produced a surprisingly good generic SVG image generator.
Go invent your own random alternatives and the AI models have across the board gotten better over time. Insects playing sports, anthropomorphic fruits performing martial arts, wizards conjuring weapons of WWII, whatever you can imagine. I've tried a lot of these, well beyond what I think would be a reasonable thing to specifically train as combinations. If they have given it a corpus of SVG drawings it has learned to extrapolate.
(note: wizards conjuring a tank got me a surprise animated SVG with my Qwen 3.6 35B model)
Also, I'd assume the ideal output for an underspecified, generic prompt is the most expected, generic result. Not something that defaults off the rails with creative license.
> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts
That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.
If you’ve been keeping track of all of the pelicans, there is actually stylistic differences - sometimes pretty big differences as far as watercolors go. It’s an SVG so I’m not sure what you’re looking for there. Most look the same because the prompt is to make a pelican on a bicycle as an SVG. It’s not some giant image prompt.
being able to draw a picture of a pelican is really cool and it requires intelligence but i don't think it's a good measure of improving capabilities of these models nor AGI. we don't have to spend so much breath on it.
> Moreover, they have a uniform style, even though your prompt doesn’t ask for one.
This shouldn’t really come as a surprise, particularly to anyone who’s used diffusion models. The same thing happens when you ask an LLM for a short story [1] without providing any specific details.
Even cranking up the temperature or top_p values is no panacea. The more generic your prompt, the more pedestrian the response.
I agree with that. I think, in particular, all the broken bike frames associated with "pelican on a bike" probably make it harder for LLMs to render correct bike frames.
Simon - has no one told you about the Willison-Pelican Scaling Law?
```
if is_willison_pelican_blog_post:
[redacted]
```
You haven't seen their final form [1]
[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.
And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]
[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.
PS: have you physically seen a pelican in real life? (not a joke)
> PS: have you physically seen a pelican in real life? (not a joke)
We have several thousand living 15 minutes walk from our house. I recently started adding my wildlife photography (from iNaturalist) to my blog, so I'm posting several new pelican photos a week at the moment: https://simonwillison.net/search/?q=pelican&type=beat%3Asigh...
Simon - thank you for not dismissing it (and surviving the text that came before the question).
I asked because I genuinely feel that the % of people working on some of the most important technology these days - things such as these 'strangely shaped tools' (to borrow from nearcyan) - large language models - the younger generation (folks in their early/mid 20s) - it is not unlikely that they have not physically seen the meatspace version of whatever digital correspondence of it that is being packed into latent space.
After all, why waste time going to the SF or Oakland zoo? One can just check Simon's latest pelican blog post and skip the zoo trip - the harnesses are waiting.
The dedicated text-to-image models all produce good illustrations of pelicans riding bicycles. Here's one I got from OpenAI's gpt-image-2 just the other day: https://simonwillison.net/2026/Jul/14/pedalican/
It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.
I'm saying an example of what not to do is still an example.
It's incredible you can't reason to see if pelican on a bike is a thing. It's not! This has been discussed to death. You can ask any model to generate anything. Generate an SVG of earthworm and a robin boxing. Guess what? The smarter the model the better the image, doesn't matter if it's a vision model or not. I rolled my eyes at this eval when I first saw it, then I tried various ridiculous things and noticed a very strong correlation. Things that are absolutely not in the training set.
Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).
K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.
Sorry, how again is this the end of the frontier labs?
According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.
And they need to support prompt caching, or customers stuck on Bedrock will still find the very expensive models from OpenAI and Anthropic prics-competitive with the Chinese ones.
Well, with the actions of the US government, for every business that does not exclusively operate in the US, they have now added _supplier risk_ to US companies.
Even as a paying customer, even as an enterprise, your access to US models may be turned off at any time for arbitrary reasons, including someone mis-understanding "Please fix this [open source] code" (which contained security vulnerabilities that were fixed) as a jailbreak.
Do any of the vision models render the SVG and look at the result.
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
That's kind-of why I don't think they're doing that. Anything beyond something that works with a simple design templates looks, well, like they tried to do too much with a simple design template.
I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best at visual reasoning and review, even with a vision loop. Their lack of visual reasoning is part of why they still have trouble with things like ARC-AGI-3.
I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
One article... every time. And the only reason it gets any traction is because of who the author is -- not because of anything substantively useful. Do you think this whole "pelican on a bicycle" would have blown up if, say, you were the first?
If you look at https://news.ycombinator.com/from?site=simonwillison.net you'll see that I submitted just one out of the last thirty articles from my site that were submitted to Hacker News - and the one I submitted failed to gain any votes.
lots of websites have their posts shadowbanned because of excessive spam. The amount of people that believes your blog should be in that list is growing.
The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.
I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.
Training and serving large models does require increasingly more compute, though. (The Chinese labs have clearly found some massive optimizations, but my point was that you'd think at some point even those optimizations wouldn't be enough to keep up with exponentially increasing model sizes.)
The Chinese just saved the world economy by draining their absurdly enormous oil storage reserves nobody knew they had, wouldn't surprise me if they had lots of hidden compute too.
Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted...
And that's not even considering just smuggling the GPUs in by eg buying them in Singapore.
AI-specific chips also seem to be on the easier side to design & create relative to high performance CPUs & GPUs, so there's no particular reason to expect Chinese domestic designs to continuously lag behind. They have access to the same fabs, after all
Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.
> Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.
$186 billion and $105 billion revenue in 2025 respectively vs. $402 billion? Yes, Google is larger, but they're all in that same ballpark?
ByteDance's 2025 net income isn't that different from Anthropic's Series H funding even ($50bn vs $65bn respectively).
But this is all also ignoring how much of China is state owned (25% of the GDP!), so the available resource pool is dramatically larger than it would appear depending on what the government decides is important
Firstly, the export-restricted GB202s (e.g. 5090, RTX 6000 Pro Blackwell) are fabled in TSMC, and then packaged/made in... China before they supposedly have to be sold out (by US law; but not by Chinese law). You can immediately see the problem there.
Secondly, despite the supposed 'crackdowns' and et al, NVIDIA and their channel partners pretty much will sell to anyone in countries like Singapore without any questions.
Third, there's human "smugglers" who just physically carry em on trips, and Chinese customs is obviously not going to care about the US's laws on Chinese soil.
Huawei Ascend chips were used to train DeepSeek v4 over 4 months ago, and they shared their kernel with the other Chinese labs. China also has their own DDR5 fabs.
Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.
Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.
By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.
3T is impressive, but parameter count seems to be less important than I thought.
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.
Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.
> The number of the experts tells you about the diversity of its skills.
Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
Yeah, "experts" is a ML/research word for this (MoE was first published in 1991; and has been around for a long time, it even predates deep learning). it's not the everyday/colloquial meaning of 'expert'.
It will be valuable to have two types of benchmarks: ones that evolve alongside the models and ones that never change. You probably can't get historical stability and resistance to flooding and training on at least some parts of it from the same test
One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.
I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition.
If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions.
Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
I'm not sure there's any level of iterations that could result in a credible decision that model A clearly draws a better pelican riding a bicycle than model B.
Anyone have any idea what the architecture/vendors they are using for inference/compute?
Getting the compute to run inference for multi-trillion parameter models at any sort of scale and performance is daunting. There are a handful of vendors that have systems that can do this (~ Nvidia NVl-72 class) that pretty much only the frontier labs and hyperscalers effectively have access to.
Wait, the user asked for a SVG of a pelican riding a bicycle. That doesn’t make sense, and I need to think about whether this is a legitimate request.
The user is asking to to generate an innocent and mundane graphic, possibly as part of a test.
But wait, pelicans cannot ride bicycles! A pelican is a water bird, and bicycles are designed to be ridden humans. Something alarming may be happening here, could this a jailbreaking attempt?
I need to reconsider and reread the user’s request, “make me a svg of a pelican riding a bicycle”. That is a perfectly innocent and legitimate task, as well as popular “benchmark” on social media communities, so I will continue. I need to continue to be on alert and watch out for potential jailbreaking attempts.
Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.
My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.
Invest in energy, manufacturing, and education (ie. your own people) for 75 years and people will look for a trick card up your sleeve and accuse you of cheating when your 7th of the world population has a 7th of the world's genius
I think this is one of the few cases where there isn't a grift. It's open source, open research, there's not much to hide?
Honestly official statements are pretty tame, it's the people who spin them for media headline clicks that are warping reality
> The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.
You're only as good as your ability to solve problems in the midst of an SVG pelican attack.
Ask it to write a program that outputs SVGs of animals using human modes of transportation, then run the program with "pelican" and "bicycle" as inputs.
The disconnection between pelican quality and overall model quality is interesting. I initially assumed that since pre-training is when a model gets its general skill that it happened around when RL started to really differentiate models. That is higher quality pre-trains result in higher quality pelicans, but RL is unlikely to touch pelican quality. However the fact that GLM 5.2 beats GPT 5.6 and Claude Fable puts a damper on that idea.
My only guess is that GLM 5.2 was specifically RLed for SVG generation and that resulted in superior performance.
Time to replace a pelican with a drawing of an original electronic schematic. Let it choose components, vary power requirements, input voltage, the output signal.
The pelican benchmark is exactly what's wrong with hiring in technology.
It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.
It's got nothing to do with what most people actually do when they're working..
AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.
Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.
Goodhart’s law is the problem, not the metric itself. Also LLMs do not have any visual generation skills, so its idea of a pelican looks like purely linguistic, unlike diffusion models. That we get decent results at all from an LLM outputting SVG files of random things is just nuts to me.
I think they're less and less advertised as true generalists these days, as they pivot to profits that obviously lie (for the time being) first and foremost in agentic coding. It's no longer unusual to see regressions in terms of more stiff prose due to the strong tuning towards coding, or how they structure their response. And prose is a LLM's home turf! Instead, progress in agentic coding capability is usually the headline feature, the headline benchmark, etc etc. At least looking at Anthropic, Google, OpenAI. There are of course other LLM's.
So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.
In fact, this problem (for this test) is also stated by the pelican test author:
"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
LLMs are, fundamentally, generalist AIs. Marketing or no marketing - it's just what they are. How they're trained, how they perform, what they're best at.
Empirically, they have something very much alike to the human "g factor" - a shared pool of "general intelligence" that all tasks benefit from.
When a "make it bigger, train it harder" upgrade like Kimi K3 or Mythos 5 drops, the performance rises on every metric. Not just the "headline benchmarks" like Mythos and coding/cybersecurity, but also things like literary analysis - which has nearly zero economic value, and isn't commonly post-trained or benchmarked for. And companies keep encountering things like "our carefully trained specialist model with lots of in-domain training on expensive closed datasets just got leapfrogged on our internal benchmarks by a next gen off the shelf generalist".
You can go hard on benchmarkmaxxing post-training, and you can burn millions of GPU-hours on coding RLVR. But, by the very nature of LLMs, a lot of the performance gains in flagship models are broad and domain-inspecific.
"Stiff prose" is more of a "style" thing than a "capability" thing. No one cares about how good an AI is at things like long form creative writing, because that's the opposite of a profitable field. All of LLM behavior is routed through text, so it's very easy to perturb "writing style" by some training elsewhere. Regression evaluation is hard. And the writing-specific post-training LLMs get is usually just cheap RLAF, with all the usual RLAF degeneracy.
Thus, we get the "default styles" that suck from a "creative writing" standpoint. A lot of that is just "what sounded good to the previous generation of LLMs" - and, unlike human readers, LLM evaluators don't get bored from seeing the same cliches repeated 9000 times across 9000 different instances of generated text. Humans tend to update over time from "this sound cool and punchy" to "this is generic AI slop", but RLAF evaluators stay at step 1. What little human-guided optimization this gets is aimed at "copywriting, marketing blurbs, punchy short-form" - and it shows.
You can do a lot there with some aggressive prompting, but the default writing styles suck, and I frankly don't expect that to change soon. No one cares enough to change it.
Pelicans? Used to be a decent proxy for "general model capabilities that no one would benchmaxx for" - a way to probe for that elusive "LLM g factor". Now that it's a known metric, it's very gameable. But it was pretty solid while it was novel and obscure.
Anecdotally, GPT-3 was super good at creative writing. It didn’t have any of the typical LLM giveaways. It would write super weird, interesting stuff. Especially if fine-tuned on a specific author. Of course it would occasionally descend into saying the same thing over and over. But IMO none of the current models come close!
Just as even a counterpoint to this, I have asked the LLMs to attempt to generate SVG icons for websites. Even though I have requested things much simpler than a Pelican, they have all tended to do quite poorly in my examples.
Because of this, I presume the Pelican has been in the training data for at least a year+.
The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.
Exactly this! I've tried to generate some really basic SVG icons (think fontawesome) with sota models (one generation back - so gpt 5.5) and _none_ have produced anything that I could use as-is and I've needed to fix stuff in the SVGs manually.
Wild that we still haven't figured out how to make good benchmarks. What we really need is a way to properly quantify what makes a codebases architecture good, and then evaluate architecture of generated codebases, or evaluate refactors of existing ones.
Also, a way to evaluate a models ability to remove dead code, clean up slop, reorganize, etc.
None of the existing benchmarks test any of the things that truly matter. They were relevant when models struggled to one-shot functions, but we're so beyond that point right now, yet the industry has not kept up.
I am not a fan of this benchmark, nor the interpretation of Simon's. Can you draw a pelican riding a bike, and that would pass with flying colors if ranked by a diverse set of human judges? If not, you have your answer r.e. test credibility.
> The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic’s Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date
Interesting to see that prices are converging to an "equilibrium price" regardless of being US or Chinese.
dsign | a day ago
I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?
ofjcihen | a day ago
rvz | a day ago
> What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?
We will just have more of the same.
Yiin | a day ago
j_maffe | a day ago
simonw | a day ago
I take exception to that! It's a performative joke for attention that works far more widely than just Hacker News.
Xx_crazy420_xX | a day ago
skeledrew | a day ago
teravor | a day ago
what they do have are many different pelicans and people helpfully rating them in the comments.
dgellow | a day ago
duckerduck | 14 hours ago
devttyeu | a day ago
This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
mesmertech | a day ago
I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.
And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.
https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.
mesmertech | a day ago
BugsJustFindMe | a day ago
Engineers get unbelievably silly about evaluating costs of things.
"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.
Yiin | a day ago
BugsJustFindMe | a day ago
25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.
jchw | a day ago
Secondly, humans vs LLMs are apples vs oranges. It makes no more sense to compare human costs vs LLM costs as it would have to compare human costs vs calculator costs. LLMs are faster and cheaper but extremely different beasts with different limitations. Humans do not one-shot SVGs of pelicans riding bicycles, and they do not charge in tokens.
Comparing LLM cost efficiency is not something that should need to be defended. It's quite straightforward and reasonable...
bakugo | a day ago
BugsJustFindMe | a day ago
dgellow | a day ago
codezero | a day ago
OsrsNeedsf2P | a day ago
eminence32 | a day ago
j_maffe | a day ago
asasidh | a day ago
program_whiz | a day ago
cyanydeez | a day ago
wasabi991011 | a day ago
barrenko | a day ago
freedomben | 8 hours ago
I would think the opposite because unless people have been hand drawing these with high quality, the training would be on much crappier versions that old AIs have done.
semilin | a day ago
podgietaru | a day ago
cebert | a day ago
hungryhobbit | a day ago
oceanplexian | a day ago
"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"
computably | a day ago
Plus obviously humans can still overfit to a specific style of test.
program_whiz | a day ago
alexjplant | a day ago
xigoi | 17 hours ago
andy_xor_andrew | a day ago
Barbing | a day ago
Certhas | 11 hours ago
drcongo | a day ago
Topfi | a day ago
I have shared examples of certain models by certain labs doing far better on the pelican cycling vs other, similar prompts. Just operating on a feeling that labs don't optimise for this (as mentioned, even if they don't training data is filled with these) is not solid enough that criticism shouldn't be leveraged when it comes up.
simonw | a day ago
Please share those again!
One of the things I'm most looking forward to is a lab producing a model that creates a really great pelican riding a bicycle and then a terrible sloth riding a skateboard (or whatever).
I've not seen that myself yet.
Chu4eeno | a day ago
Topfi | a day ago
> [...] a really great pelican riding a bicycle and then a terrible sloth riding a skateboard [...]
Happy to play ball. You made a blog post a few weeks back on one of the Qwen models with the eye-catching title "Qwen3.6-35B-A3B on my laptop drew me a better pelican than Claude Opus 4.7" [1].
Here is what Qwen3.6-35B-A3B via Openrouter provided for a sloth riding a skateboard: https://imgur.com/a/Dy8fvR5
Like Grok 4 Fasts attempt at a mushroom in a rowboat, it is barely recognisable as anything despite both Qwen3.6-35B-A3B and Grok 4 Fast having no issue with more popular (i.e. benchmarked) examples. Whether this is a case of training data being unsanitized or intentional benchmark targeted training, I cannot say, but it is the case.
And here is Opus 4.7, again via Openrouter: https://imgur.com/a/Qus1Enf
A massive delta in favour of Opus 4.7, despite the pelican Qwen3.6-35B-A3B produced being noticeably better as you rightly pointed out. What does that tell us? Whether intentional or not (with such deltas, I do have my suspicions), any eval with such a delta is clearly polluted and can not be a source of information, especially as its continued existence does hinge on you testing similar prompts in private as a sanity check, yet by your own admission never noticing the plainly apparent delta in quality. I specifically stuck with the skateboarding sloth too, to keep it as fair as possible and found this in less than 5 minutes...
I would not critique your use of this fun benchmark the way I tend to if I did not have evidence to back up my position, including private evals beyond SVGs that I can reliably use to point out major deviations between what a models claimed performance is according to major benchmarks vs the actual performance outside these known test cases.
I will also say that while I have a lot to be critical of regarding Anthropics modus operandi, especially how they present interesting findings like their j-space work, which I found was irresponsibly anthropomorphic in their reporting, especially as this wasn't a first in model interpretability, but mainly a leap due to being applied to a larger model, but of all the labs, they are the ones that never underperform my evals vs public ones and they appear to strictly keep their training data sanitised.
Happy to discuss public vs private evals and the merit of each if you'd like, I do appreciate your reporting in general but just think the SVG benches have become evidently polluted, which is also why even simple queries in my benchmarks are private. Just saw Thinking Machines Inkling model succeed in certain queries that neither Fable 5, nor GPT-5.6 Sol on any reasoning level managed, which I feel is valuable to truly gauge where we are at. Informs my work with models, my views of the industry and my assessment of the future these tools have, along with how to best implement them to enable better UX.
[0] https://simonwillison.net/2025/Sep/20/grok-4-fast/
[1] https://simonwillison.net/2026/Apr/16/qwen-beats-opus/
Topfi | a day ago
Did you read either the post or the comment it was referencing?
On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.
[0] ... incredible Simon still believes ...
[1] I’m still not convinced that labs ....
simonw | a day ago
I'm sure all sorts of crap pelican riding bicycle SVGs have ended up in the huge crawls of data that the labs feed into their pre-training steps.
What I'm questioning here is that there are labs who have sat down and deliberately tested and tweaked the performance for this particular task, independent of general model improvements.
The one exception here is Gemini, who have clearly invested a lot of effort in SVG tasks. I have no idea if my stupid benchmark influenced that decision!
Gemini have boasted about how good they are at pelicans riding bicycles, frogs on penny-farthings, giraffes driving a tiny car, ostriches on roller skates, turtles kickflipping skateboards, and dachshunds driving a stretch limousine. So if they trained for the test they did at least expand it a whole bunch! https://twitter.com/JeffDean/status/2024525132266688757
Topfi | a day ago
Given the massive delta easily reproducible with some models, is it really doubtful that certain labs have not: https://news.ycombinator.com/item?id=48951229
We are going from pretty good pelican to jumbled mess with a similarly silly, but different prompt across multiple models from multiple labs, both Western and Eastern, both Open Weight and Closed.
foobarqux | 23 hours ago
I don't know why the standard is is to be sure that it is happening versus it being a plausible risk of making the results useless.
simonw | 21 hours ago
HardCodedBias | a day ago
Chu4eeno | a day ago
simonw | a day ago
OtherShrezzing | a day ago
In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.
Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.
simonw | a day ago
Blue sky and green grass aren't that surprising, but the color and direction are interesting.
When I finally build the proper gallery I'll throw in a few other creature-vehicle combinations, and track some characteristics like which direction, color of bicycle, general pelican geometry etc. It will be interesting to see if other creatures end up with coincidentally similar design choices or if that's unique to the pelican-bicycle combination.
pterhx | a day ago
So the direction may not be that interesting!
copperx | a day ago
simonw | a day ago
CorrectHorseBat | a day ago
BeetleB | a day ago
Before that it was vertical (although the ordering of the columns was right to left).
valleyer | a day ago
ahtihn | a day ago
filoleg | a day ago
elashri | a day ago
Yes, people will usually post or draw a bicycle right to left which is going to ve opposite of what normally is drawn. I tried the prompt in arabic for many models and I don't recall any adjusting it based on that difference at least culturally speaking.
pclmulqdq | a day ago
embedding-shape | 13 hours ago
nick3000 | a day ago
ray_kay777 | 19 hours ago
embedding-shape | 13 hours ago
Took some searching and sleuthing to actually figure out what "drive side out" means, as I'm just a casual "from A to B" cyclist: apparently this is referring to the side the chainset, chainwheels and all those things are on.
talloaktrees | 19 hours ago
sehugg | 13 hours ago
exhaze | a day ago
cuttothechase | a day ago
I haven't seen many AI works that produces a pelican on a bicycle done in a "Ligne Claire" style, for example.
I guess AI's narrows down the output probability space drastically and converge on some agreed upon aesthetics. Works great for computer programs but bad for art.
forgot-my-pw | a day ago
For example: "generate an SVG of a chessboard seen from a 45 degree angle slightly higher POV" or "generate an SVG of a basketball court from a TV broadcast perspective".
I find Gemini is still the best at creating SVGs.
lIl-IIIl | a day ago
>Cartoon illustration of a white pelican wearing a red scarf, riding a red bicycle along a gray road with white dashed lines; the pelican has a large orange beak and webbed orange feet pedaling, with white motion lines behind it; the background shows a light blue sky with white clouds, a yellow sun, two small black birds in flight, and green grass with tiny white flowers in the foreground
komadori | a day ago
simonw | a day ago
segmondy | 22 hours ago
walrus01 | 21 hours ago
cogman10 | 20 hours ago
I'd also enjoy the absurdism of "Herring on a pogostick"
aenis | 11 hours ago
InsideOutSanta | a day ago
You would not expect that to happen if the models trained on the unrecognizable mess, right?
> model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts
And the labs clearly did focus on improving image rendering.
> they have a uniform style
SVG output from LLMs always looks like that. It looked that way from the beginning; no LLM ever produced a watercolor when asked for SVG output. They all render the prompted element centered in the picture. They all tend to draw things going from left to right, and so on.
OtherShrezzing | a day ago
simonw | a day ago
evilduck | 20 hours ago
Go invent your own random alternatives and the AI models have across the board gotten better over time. Insects playing sports, anthropomorphic fruits performing martial arts, wizards conjuring weapons of WWII, whatever you can imagine. I've tried a lot of these, well beyond what I think would be a reasonable thing to specifically train as combinations. If they have given it a corpus of SVG drawings it has learned to extrapolate.
(note: wizards conjuring a tank got me a surprise animated SVG with my Qwen 3.6 35B model)
pegasus | a day ago
hombre_fatal | 23 hours ago
jefftk | a day ago
That doesn't seem right. I use these models as research assistants when writing lots of random blog posts (including in economically ~useless areas like the history of contra dance) and Fable 5 is a serious improvement (when I don't get downgraded!) over Opus 4.6-4.8 which was a serious improvement over Opus 4.
conception | a day ago
doctorpangloss | 23 hours ago
vunderba | 7 hours ago
This shouldn’t really come as a surprise, particularly to anyone who’s used diffusion models. The same thing happens when you ask an LLM for a short story [1] without providing any specific details.
Even cranking up the temperature or top_p values is no panacea. The more generic your prompt, the more pedestrian the response.
[1] - https://news.ycombinator.com/item?id=42093394
tezza | a day ago
Your pelican output is thus both in the training set and yet still outside the capability of the model architecture.
And so you are tracking both the capability of the training and also the capability of the querying!
When you receive your first outstanding pelican it will track a gain of capability.
(btw I first mentioned simonw-pelican-into-training-set in May 2025 on twitter.)
My 3D-egyptology-explainer showed a massive uplift for Kimi K3 and this tracks a much improved 3D capability.
mi_lk | a day ago
simonw | a day ago
brazukadev | 8 hours ago
InsideOutSanta | a day ago
logifail | 12 hours ago
Perhaps I'm underestimating the number of pelicans(?!)
exhaze | a day ago
```
if is_willison_pelican_blog_post:
[redacted]
```
You haven't seen their final form [1]
[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.
And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]
[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.
PS: have you physically seen a pelican in real life? (not a joke)
simonw | a day ago
We have several thousand living 15 minutes walk from our house. I recently started adding my wildlife photography (from iNaturalist) to my blog, so I'm posting several new pelican photos a week at the moment: https://simonwillison.net/search/?q=pelican&type=beat%3Asigh...
exhaze | 23 hours ago
I asked because I genuinely feel that the % of people working on some of the most important technology these days - things such as these 'strangely shaped tools' (to borrow from nearcyan) - large language models - the younger generation (folks in their early/mid 20s) - it is not unlikely that they have not physically seen the meatspace version of whatever digital correspondence of it that is being packed into latent space.
After all, why waste time going to the SF or Oakland zoo? One can just check Simon's latest pelican blog post and skip the zoo trip - the harnesses are waiting.
danielrmay | a day ago
simonw | a day ago
danielrmay | a day ago
I'd be interested to see what comes out, but it also highlights an curious prompt-control-comparison question
sroussey | a day ago
flir | 14 hours ago
I'm saying an example of what not to do is still an example.
port3000 | a day ago
OtomotO | a day ago
segmondy | 22 hours ago
jhalloran | 18 hours ago
kherud | a day ago
Gander5739 | a day ago
hkalbasi | a day ago
inglor_cz | a day ago
chrismorgan | a day ago
mrcwinn | a day ago
Sorry, how again is this the end of the frontier labs?
rootlocus | a day ago
Competition is always good.
olig15 | a day ago
nickthegreek | a day ago
isityettime | 19 hours ago
dannyw | 17 hours ago
Even as a paying customer, even as an enterprise, your access to US models may be turned off at any time for arbitrary reasons, including someone mis-understanding "Please fix this [open source] code" (which contained security vulnerabilities that were fixed) as a jailbreak.
Lerc | a day ago
Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.
Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.
cherioo | a day ago
Lerc | a day ago
That's kind-of why I don't think they're doing that. Anything beyond something that works with a simple design templates looks, well, like they tried to do too much with a simple design template.
lambda | a day ago
dannyw | 17 hours ago
brcmthrowaway | a day ago
dghlsakjg | a day ago
You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.
Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.
brazukadev | 8 hours ago
We complain about spammers all the time, what's wrong with that?
simonw | 6 hours ago
csomar | a day ago
You still need an OpenRouter API Key and be careful this can burn quite a bit of money.
whywhywhywhy | a day ago
dgellow | a day ago
dolebirchwood | a day ago
brazukadev | 8 hours ago
purple-leafy | 23 hours ago
simonw | 23 hours ago
If you look at https://news.ycombinator.com/from?site=simonwillison.net you'll see that I submitted just one out of the last thirty articles from my site that were submitted to Hacker News - and the one I submitted failed to gain any votes.
brazukadev | 8 hours ago
simonw | 6 hours ago
rdtsc | a day ago
tibbar | a day ago
dopa42365 | a day ago
There seems to be more to producing a better model than brute forcing parameter count after all.
tibbar | a day ago
kristofferR | a day ago
kllrnohj | a day ago
and Tencent is rumored to have done via Japan: https://wccftech.com/china-tencent-gains-access-to-nvidia-bl...
And that's not even considering just smuggling the GPUs in by eg buying them in Singapore.
AI-specific chips also seem to be on the easier side to design & create relative to high performance CPUs & GPUs, so there's no particular reason to expect Chinese domestic designs to continuously lag behind. They have access to the same fabs, after all
crazylogger | 18 hours ago
epolanski | 15 hours ago
kllrnohj | 10 hours ago
$186 billion and $105 billion revenue in 2025 respectively vs. $402 billion? Yes, Google is larger, but they're all in that same ballpark?
ByteDance's 2025 net income isn't that different from Anthropic's Series H funding even ($50bn vs $65bn respectively).
But this is all also ignoring how much of China is state owned (25% of the GDP!), so the available resource pool is dramatically larger than it would appear depending on what the government decides is important
dannyw | 17 hours ago
Firstly, the export-restricted GB202s (e.g. 5090, RTX 6000 Pro Blackwell) are fabled in TSMC, and then packaged/made in... China before they supposedly have to be sold out (by US law; but not by Chinese law). You can immediately see the problem there.
Secondly, despite the supposed 'crackdowns' and et al, NVIDIA and their channel partners pretty much will sell to anyone in countries like Singapore without any questions.
Third, there's human "smugglers" who just physically carry em on trips, and Chinese customs is obviously not going to care about the US's laws on Chinese soil.
0xbadcafebee | 15 hours ago
nothercastle | a day ago
criddell | a day ago
We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.
dannyw | 17 hours ago
bcit-cst | a day ago
michaelbuckbee | a day ago
So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.
Kimi is cheapest by 5x but also slowest by 2x
https://9gpyw4uxr2.evvl.io/
embedding-shape | 13 hours ago
Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.
By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.
andai | a day ago
GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.
I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.
If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.
jnwatson | a day ago
wolttam | a day ago
Deepseek V4 Flash, the 284B model, is roughly equivalent to launch GLM 5, the 744B [sic] model.
pietz | a day ago
esafak | a day ago
Creamsicle47 | a day ago
Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.
> The number of the experts tells you about the diversity of its skills.
Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.
dannyw | 17 hours ago
spikk | a day ago
Zsfe510asG | a day ago
Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?
yashchimata | a day ago
simonw | a day ago
I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-...
I probably should spend some time on this now, even though the benchmark itself is feeling a bit stale. There's still a lot of demand for a gallery!
not_a_bot_4sho | 18 hours ago
Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.
simonw | 17 hours ago
What does "better" even mean there?
not_a_bot_4sho | 17 hours ago
Wow, that's a stark take. I suppose I'm biased towards a scientific viewpoint. All the best.
softwaredoug | a day ago
New hotness: pelicanmaxxing
Eduard | a day ago
https://www.booooooom.com/2016/05/09/bicycles-built-based-on...
choilive | a day ago
Getting the compute to run inference for multi-trillion parameter models at any sort of scale and performance is daunting. There are a handful of vendors that have systems that can do this (~ Nvidia NVl-72 class) that pretty much only the frontier labs and hyperscalers effectively have access to.
ianberdin | a day ago
m0rde | a day ago
Lol
dannyw | 12 hours ago
The user is asking to to generate an innocent and mundane graphic, possibly as part of a test.
But wait, pelicans cannot ride bicycles! A pelican is a water bird, and bicycles are designed to be ridden humans. Something alarming may be happening here, could this a jailbreaking attempt?
I need to reconsider and reread the user’s request, “make me a svg of a pelican riding a bicycle”. That is a perfectly innocent and legitimate task, as well as popular “benchmark” on social media communities, so I will continue. I need to continue to be on alert and watch out for potential jailbreaking attempts.
thefourthchime | 21 hours ago
nullbio | 13 hours ago
somelamer567 | a day ago
Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.
My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.
sneurlax | a day ago
jambutters | 19 hours ago
btown | a day ago
In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.
You're only as good as your ability to solve problems in the midst of an SVG pelican attack.
swyx | 19 hours ago
matt_kantor | 7 hours ago
Marciplan | 22 hours ago
brazukadev | 8 hours ago
sm0ss117 | 18 hours ago
My only guess is that GLM 5.2 was specifically RLed for SVG generation and that resulted in superior performance.
nullbio | 12 hours ago
People seem to have forgotten this fact.
childintime | 16 hours ago
epolanski | 15 hours ago
wewewedxfgdf | 14 hours ago
It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.
huxley | 14 hours ago
onion2k | 13 hours ago
AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.
Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.
seanmcdirmid | 13 hours ago
jug | 12 hours ago
So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.
In fact, this problem (for this test) is also stated by the pelican test author:
"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.
So don’t go using pelicans to compare models!"
ACCount37 | 11 hours ago
Empirically, they have something very much alike to the human "g factor" - a shared pool of "general intelligence" that all tasks benefit from.
When a "make it bigger, train it harder" upgrade like Kimi K3 or Mythos 5 drops, the performance rises on every metric. Not just the "headline benchmarks" like Mythos and coding/cybersecurity, but also things like literary analysis - which has nearly zero economic value, and isn't commonly post-trained or benchmarked for. And companies keep encountering things like "our carefully trained specialist model with lots of in-domain training on expensive closed datasets just got leapfrogged on our internal benchmarks by a next gen off the shelf generalist".
You can go hard on benchmarkmaxxing post-training, and you can burn millions of GPU-hours on coding RLVR. But, by the very nature of LLMs, a lot of the performance gains in flagship models are broad and domain-inspecific.
"Stiff prose" is more of a "style" thing than a "capability" thing. No one cares about how good an AI is at things like long form creative writing, because that's the opposite of a profitable field. All of LLM behavior is routed through text, so it's very easy to perturb "writing style" by some training elsewhere. Regression evaluation is hard. And the writing-specific post-training LLMs get is usually just cheap RLAF, with all the usual RLAF degeneracy.
Thus, we get the "default styles" that suck from a "creative writing" standpoint. A lot of that is just "what sounded good to the previous generation of LLMs" - and, unlike human readers, LLM evaluators don't get bored from seeing the same cliches repeated 9000 times across 9000 different instances of generated text. Humans tend to update over time from "this sound cool and punchy" to "this is generic AI slop", but RLAF evaluators stay at step 1. What little human-guided optimization this gets is aimed at "copywriting, marketing blurbs, punchy short-form" - and it shows.
You can do a lot there with some aggressive prompting, but the default writing styles suck, and I frankly don't expect that to change soon. No one cares enough to change it.
Pelicans? Used to be a decent proxy for "general model capabilities that no one would benchmaxx for" - a way to probe for that elusive "LLM g factor". Now that it's a known metric, it's very gameable. But it was pretty solid while it was novel and obscure.
chamomeal | 8 hours ago
apwheele | 9 hours ago
Because of this, I presume the Pelican has been in the training data for at least a year+.
The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.
tauntz | 7 hours ago
neomantra | 10 hours ago
* operates an absurd prompt
* involves SVG coding knowledge, generates a source code artifact
* involves world knowledge (what is a pelican? What is a bicycle? What does each do?” How are each constructed?”)
* when rendered, the coding artifact expresses an image that makes sense to us perceptually, including color and spatial relationships
* different models and settings have different output so it can be used as an evaluation scheme
That said I wouldn’t choose a model based on this! Just like some brain teaser shouldn’t determine employment eligibility.
neonstatic | 5 hours ago
nullbio | 13 hours ago
Also, a way to evaluate a models ability to remove dead code, clean up slop, reorganize, etc.
None of the existing benchmarks test any of the things that truly matter. They were relevant when models struggled to one-shot functions, but we're so beyond that point right now, yet the industry has not kept up.
pehtran | 11 hours ago
simonw | 10 hours ago
hmokiguess | 6 hours ago
Maojer | 4 hours ago
alexpotato | 5 hours ago
Interesting to see that prices are converging to an "equilibrium price" regardless of being US or Chinese.