Kimi K3, and what we can still learn from the pelican benchmark

https://simonwillison.net/2026/Jul/16/kimi-k3/

by droidjj • 5 days ago

406 222 5 days ago

dsign• 5 days ago

Another day, another model and another pelican :-)

I can't help but wonder where is the trend going? What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing? Or maybe the prompt then will be "make a pelican ride a bicycle", and out will come the genetic code for a giant pelican with extremities suitable for a handle bar and pedals, and an inborn affinity to ride bicycles?

ofjcihen• 5 days ago

I’m excited for this specific brand of survival horror.

rvz• 5 days ago

You are thinking too hard on this. This entire "benchmark" is a performative joke for attention that only works on HN.

What will we have in five years? Maybe it will all have puttered out, and we will have moved to the next thing?

We will just have more of the same.

Yiin2• 5 days ago

You say it's performative joke, but it all depends what you're using model for. So far the rule has been quite straightforward, better models consistently renders pelican in higher quality, I've yet to see an exception. It is also a good enough (for me at least) test for "taste" the model has.

simonw19• 5 days ago

This entire "benchmark" is a performative joke for attention that only works on HN.

I take exception to that! It's a performative joke for attention that works far more widely than just Hacker News.

Xx_crazy420_xX• 5 days ago

I would be surprised if pelican svgs are not part of the training corpus rn

skeledrew• 5 days ago

If that were the case then it'd do a way better job. Think experienced artist level.

teravor• 5 days ago

how would great pelicans make their way into the training set?

what they do have are many different pelicans and people helpfully rating them in the comments.

dgellow3• 5 days ago

That’s covered in the article

duckerduck• 5 days ago

As mentioned elsewhere, the benchmark introduces bad pelicans in the training set. What I'm curious about however, if it's possible for a human artist to "poison" the benchmark by releasing some really good pelicans svgs and have all future models output their version.

devttyeu• 5 days ago

How does the prompt “Generate an SVG of a pelican riding a bicycle” add up to 95 input tokens? OpenAI’s tokenizer counts 10, Anthropic’s counts 10 for Opus 4.6, 30 for Opus 4.7 and 25 for Sonnet 5/Fable 5. Prompting “hi” to Kimi K3 counted 86 tokens, suggesting there may be an 85 token hidden system prompt. It refused to leak it though.

This is quite possibly reasoning-effort prompt which is injected before the opening <think> token whenever you set a custom reasoning effort, see e.g. DeepSeek-V4 max mode prompt: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...

mesmertech2• 5 days ago

My personal benchmark for new models has been to compare video making skills with something like remotion. Usually reveals if they have any "taste" or outside the box thinking.

I'm starting to not trust any "benchmarks" when it comes to frontier models at least. As an example Sol feels the most "gets stuff done" but has zero taste, or any capability to surprise.

And for frontier models I go one step ahead and try to recreate a complex animation video, with the ability for the model to review its own work. And at this Fable is still the top one. Ex: https://www.youtube.com/watch?v=uDAeAuYyl0E (recreation of Claude announcement video) and https://www.youtube.com/watch?v=cSsVNtGPOIg (recreation of a fireship video). Sol did something similar but you can instantly tell its AI slop from very small things, and it just has no narrative or thought put into the writing.

https://mesmer.tools/benchmarks/ai-video-generation , I usually put basic ones here.

mesmertech2• 5 days ago

And on creativity at least visually, Gemini 3.1 pro is somehow still up there. But its really hindered by its inability to use tool calls effectively or make a long term plan.

BugsJustFindMe3• 5 days ago

This is expensive—the pelican cost 25 cents!

Engineers get unbelievably silly about evaluating costs of things.

"The tokens are so expensive!" Oh my sweet child, how much would even the least capable human effort cost? This is what the executives properly understand that the programmers don't.

Yiin2• 5 days ago

they're comparing to similar capability llm models, not humans. If one dishwasher does job at similar quality as another dishwasher, but using 30% more water and energy, you wouldn't compare to how much it costs human to do the same work, it would make no sense.

BugsJustFindMe3• 5 days ago

they're comparing to similar capability llm models, not humans

25 cents is 10x the cost of 2.5 cents, but it's still extremely cheap for the product. It's very much the wrong comparison for a world where the primary competition is still humans who need to eat, and it treats percentage differences as more important than absolute differences when the opposite is true.

bakugo• 5 days ago

Would anyone pay a human to create an SVG of a pelican riding a bike?

BugsJustFindMe3• 5 days ago

In fact humans get paid to create SVGs of all kinds of things.

codezero• 5 days ago

Well, no, not now they won’t.

OsrsNeedsf2P• 5 days ago

It's incredible Simon still believes pelicans on bikes aren't part of the training set, despite hundreds of them on blogs, forums, and Github. Stuff we put in our company blog shows up known by LLMs 6 months later, and we have 1000x less traffic than Simon's own website

eminence32• 5 days ago

Pelicans and bikes can be in the training set without them training for this specific benchmark.

j_maffe2• 5 days ago

Yes and that would improve its ability to draw SVGs of pelicans on bikes, no?

semilin• 5 days ago

They can be in the training set but not deliberately trained for. There may be a lot of people posting pelican svgs, but not typically because they're high quality and worth replicating.

podgietaru• 5 days ago

More to it, the actual bloody companies are using them as a reference. Maybe it’s a 3d version, not an svg - but it clearly shows they’re on the radar of these companies.

cebert• 5 days ago

Simon has stated a few times that he knows it’s possible that pelicans could be in the training sets. He also has other tests he doesn’t share publicly. He’s just a fan of pelicans.

hungryhobbit• 5 days ago

From the article it doesn't even sound like he cares about pelicans at all, and doesn't think they are a good way to compare models anymore ... but people are used to seeing the test now, and it does serve as a common "hello world" unit of work.

oceanplexian• 5 days ago

Imagine if we applied this train of logic to humans.

"That artist saw a pelican at the beach once!" [cue the outrage] "He's not a real artist, he's a cheater and produces nothing original!"

computably• 5 days ago

Except, of course, LLMs are not humans, and they do not learn or "reason" in a way which even remotely resembles humans.

Plus obviously humans can still overfit to a specific style of test.

program_whiz2• 5 days ago

This is a sight-reading test. If a musician practices a piece for thousands of hours, it would no longer be an effective sight reading / creativity test. The purpose of the test was to see how models would compose something novel requiring the ability to compose orthogonal, normally unrelated, components into a coherent image.

alexjplant• 5 days ago

We do. People who, for example, memorize question banks to pass certification tests without knowing the underlying material are equally frowned upon for not having the problem solving skills that they purport to. I'll leave the contrasts between LLMs and people to the well-written sibling comments.

xigoi• 5 days ago

More like “This artist won the drawing competition because someone told her the theme in advance and the specifically practiced drawing pelicans for hundreds of hours.”

andy_xor_andrew• 5 days ago

Did you read the post? It's not even that long. He explicitly mentions this...

Barbing• 5 days ago

Are they responding to: “I’m still not convinced that labs are training for the benchmark—if they were, I’d expect much better results.”

drcongo• 5 days ago

Clearly not. There's a subset of HN users who rush to post this same thing every single time.

Topfi4• 5 days ago

Respectfully, did you? The comment was specific to doubting the believe simonw has that labs are not training [0] specifically for this task, which is exactly what simonw wrote in the post [1], that it is a believe of his that they don't. He did not mention any kind of evidence or any piece of information that would indicate that the commenter didn't read the blog post.

Did you read either the post or the comment it was referencing?

On the note of training on SVGs, I have seen some labs models outperform when prompted for SVGs of certain animal and action combinations (pelican on bike, panda eating burger, etc.) compared to other similarly outlandish prompts for SVG output that are not part of widely reported benchmarks, even shared evidence one of the last times this came up on here.

[0] ... incredible Simon still believes ...

[1] I’m still not convinced that labs ....

HardCodedBias• 5 days ago

A person from Google famously put on her linkedin that her job was to optimize SVG for Gemini 3.0.

Chu4eeno2• 5 days ago

SVG output is useful, though. I often ask whatever LLM I have open to generate placeholder icons whenever I need them.

simonw19• 5 days ago

The pelicans are still all rubbish. If they make it into the training set it doesn't help the models produce better pelicans, if anything it will make them perform worse!

OtherShrezzing2• 5 days ago

Respectfully, the pelicans used to be an unrecognisable mess and now they’re unquestionably pelicans on bicycles, rendered poorly, from every model.

In the same timescale, model capabilities across the board have only meaningfully improved in places where the labs are focusing their training efforts.

Moreover, they have a uniform style, even though your prompt doesn’t ask for one. There's no model going rogue and producing a watercolour of a pelican. They’re all rendered in an approximately uniform style, even though the svg format has a basically unlimited possibility space.

tezza• 5 days ago

Yes, I see your point.

Your pelican output is thus both in the training set and yet still outside the capability of the model architecture.

And so you are tracking both the capability of the training and also the capability of the querying!

When you receive your first outstanding pelican it will track a gain of capability.

(btw I first mentioned simonw-pelican-into-training-set in May 2025 on twitter.)

My 3D-egyptology-explainer showed a massive uplift for Kimi K3 and this tracks a much improved 3D capability.

mi_lk• 5 days ago

At this point I am simply interested in how much longer you're gonna ride this schtick

InsideOutSanta2• 5 days ago

I agree with that. I think, in particular, all the broken bike frames associated with "pelican on a bike" probably make it harder for LLMs to render correct bike frames.

exhaze3• 5 days ago

Simon - has no one told you about the Willison-Pelican Scaling Law?

```

if is_willison_pelican_blog_post:

[redacted]

```

You haven't seen their final form [1]

[1] final form is a frontend/react/let's not talk about it, library - it caused a great deal of PTSD to me and my previous company's team due to its dogmatic preference for "we use these axioms, end of story", over practical utility - so it was quite challenging to do state of the art tasks such as nested form fields (e.g. 'user.address.personal.line-1'). The PTSD it caused made us all block out the memories, I suppose. But - it had zero dependencies. That is what mattered. It kept us going. We weren't reaching for more. We had plenty of time.

And thank god for that. Because I'd forgotten my watch in California - and this was in Tokyo [2]

[2] a joke within a joke about Jensen's Kyoto gardener story. Beautiful story, drowned out by WatchGate memes. Why can't jokes have layers? Models have trillions. If you miss 100% of the jokes you don't make, make all the jokes. Someone will laugh (eventually, maybe?) Even if it's: "this person + comedy club = full secret service detail". If someone laughs at that - at my own expense? I don't mind. They laughed. I know this is a gibberish, off-topic message - it's also a human message. I just felt we need more such things in our lives these days.

PS: have you physically seen a pelican in real life? (not a joke)

danielrmay2• 5 days ago

What does good look like?

flir• 5 days ago

It's just a gut feeling, but I think you're running a (very slow) distributed hill-climbing algorithm. LLM1 generates an SVG. You post it online, with commentary on what is good/bad about it. LLM2 consumes the SVG alongside your commentary, and produces a slightly different SVG. Rinse, repeat.

I'm saying an example of what not to do is still an example.

port3000• 5 days ago

Yeah I asked Nano Banana to make a render of our company office and was scarily accurate

OtomotO• 5 days ago

It's incredible people still discuss the pelicans... But then again, the ad just works.

segmondy2• 5 days ago

It's incredible you can't reason to see if pelican on a bike is a thing. It's not! This has been discussed to death. You can ask any model to generate anything. Generate an SVG of earthworm and a robin boxing. Guess what? The smarter the model the better the image, doesn't matter if it's a vision model or not. I rolled my eyes at this eval when I first saw it, then I tried various ridiculous things and noticed a very strong correlation. Things that are absolutely not in the training set.

jhalloran• 5 days ago

This reminded me about the news cycle last year that we were running out of training data (and how silly that was)

kherud• 5 days ago

Imagine what amazing SVG generators we could have if Simon had randomized the target image from the start (and companies wouldn't just overfit on pelicans).

Gander5739• 5 days ago

I think a pelican riding a bike is fairly random. (https://xkcd.com/221/)

hkalbasi• 5 days ago

Is there a gallery of all pelicans generated by simon over time?

inglor_cz• 5 days ago

If Simon reads this debate, I would gladly vote for such a gallery. It would belong to "digital heritage of mankind".

chrismorgan• 5 days ago

https://simonwillison.net/tags/pelican-riding-a-bicycle/ isn’t quite a gallery, but pretty close.

mrcwinn• 5 days ago

K3 is as expensive as Sonnet, not great at writing English, is handing IP back to the Chinese, and once open source will be difficult to run at scale without the compute that OpenAI and Anthropic have largely grabbed.

Sorry, how again is this the end of the frontier labs?

rootlocus• 5 days ago

According to some benchmarks has the coding capability of Opus at the price of Sonnet, supposedly will be open weights and is not subject to random trade wars with allied states.

Competition is always good.

olig15• 5 days ago

You mean the scale that AWS provides with Bedrock?

nickthegreek• 5 days ago

Bedrock needs to actually update their chinese models to the newest versions for this to matter.

dannyw6• 5 days ago

Well, with the actions of the US government, for every business that does not exclusively operate in the US, they have now added _supplier risk_ to US companies.

Even as a paying customer, even as an enterprise, your access to US models may be turned off at any time for arbitrary reasons, including someone mis-understanding "Please fix this [open source] code" (which contained security vulnerabilities that were fixed) as a jailbreak.

Lerc2• 5 days ago

Do any of the vision models render the SVG and look at the result.

Perhaps more importantly can they do that during reinforcement training. Learning how to critically analyse the appearance of what it generates would be quite useful.

Manually feeding images back to models has been hilariously bad in the past which suggests that relating something it sees to something it wrote is not an ability it is very good at.

cherioo• 5 days ago

I imagine all vision models have to do this, this being html rendering, to be able to do well in web design.

Lerc2• 5 days ago

to be able to do well in web design.

That's kind-of why I don't think they're doing that. Anything beyond something that works with a simple design templates looks, well, like they tried to do too much with a simple design template.

lambda• 5 days ago

I've tried doing a loop of rending the SVG and then tweaking based on that, with local models (so, not nearly as strong). It wasn't very successful; it would mostly report that the image looked great and didn't need any tweaks. Maybe I should try it again, there have been some newer models since I first tried it. And yeah, maybe worth trying with bigger models. But I have found that models aren't necessarily the best at visual reasoning and review, even with a vision loop. Their lack of visual reasoning is part of why they still have trouble with things like ARC-AGI-3.

dannyw6• 5 days ago

I've found much better luck giving it an audit check-list, including some steers like: are there any visual glitches or SVG bugs, are the colours consistent, etc.

brcmthrowaway• 5 days ago

Imagine shilling some CLI tools no one uses in this post.

dghlsakjg6• 5 days ago

Lighten up.

You’re reading a personal blog and complaining about an open source personal project he runs and distributes for free. He’s allowed to talk about his personal work on his personal blog. Especially considering the cli utility he talks about is directly related to the post.

Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.

brazukadev10• 4 days ago

Imagine complaining about someone generating valuable content for free and not packaging it to your personal tastes.

We complain about spammers all the time, what's wrong with that?

simonw19• 4 days ago

I think ~15,000 downloads a day is pretty good https://pypistats.org/packages/llm

csomar• 5 days ago

If anyone wants to try SVG generation from different models, I made this: https://codeinput.com/svg (here is an older generation: https://codeinput.com/s/5KEGl1e3rB3)

You still need an OpenRouter API Key and be careful this can burn quite a bit of money.

whywhywhywhy• 5 days ago

Don't see why we have to have this spammed every model release when Fable class models perform the same as Opus on basic tasks like these.

dgellow3• 5 days ago

What spam? It’s one article. You can skip it

dolebirchwood• 5 days ago

One article... every time. And the only reason it gets any traction is because of who the author is -- not because of anything substantively useful. Do you think this whole "pelican on a bicycle" would have blown up if, say, you were the first?

brazukadev10• 4 days ago

one article? more like 30 comments with a set of links to his blog.

purple-leafy• 5 days ago

I think the user should be banned. It’s insane spam

simonw19• 5 days ago

I didn't submit this story.

If you look at https://news.ycombinator.com/from?site=simonwillison.net you'll see that I submitted just one out of the last thirty articles from my site that were submitted to Hacker News - and the one I submitted failed to gain any votes.

rdtsc• 5 days ago

The idea is not to use pelicans on bikes but a similarly random non-sensical prompts: crows on scooters, squirrels in a moon rover etc. Then pick another one for another for next cross-llm evaluation.

tibbar2• 5 days ago

I wonder how the Chinese labs are training a 3 trillion parameter model on what has to be vastly smaller compute resources. If the U.S. compute advantage is persistent, it's hard to imagine that Chinese labs will be able to keep pace forever, as a matter of physics, but... so far they seem to be doing just fine.

dopa42365• 5 days ago

It's not like same parameter count models are identical, so that doesn't appear to be an indicator for quality, or even compute requirements?

There seems to be more to producing a better model than brute forcing parameter count after all.

tibbar2• 5 days ago

Training and serving large models does require increasingly more compute, though. (The Chinese labs have clearly found some massive optimizations, but my point was that you'd think at some point even those optimizations wouldn't be enough to keep up with exponentially increasing model sizes.)

kristofferR• 5 days ago

The Chinese just saved the world economy by draining their absurdly enormous oil storage reserves nobody knew they had, wouldn't surprise me if they had lots of hidden compute too.

kllrnohj2• 5 days ago

Or they just don't actually have any compute access restrictions of significance? Chinese companies can just go use those GPUs in neighboring countries that aren't export-restricted, like Malaysia. Like ByteDance openly did: https://www.tomshardware.com/pc-components/gpus/chinas-byted...

and Tencent is rumored to have done via Japan: https://wccftech.com/china-tencent-gains-access-to-nvidia-bl...

And that's not even considering just smuggling the GPUs in by eg buying them in Singapore.

AI-specific chips also seem to be on the easier side to design & create relative to high performance CPUs & GPUs, so there's no particular reason to expect Chinese domestic designs to continuously lag behind. They have access to the same fabs, after all

crazylogger• 5 days ago

Even ignoring chip export ban, Chinese companies have way less funding than American counterparts, maybe 1 or 2 orders of magnitude less depending on which company you look at. Deepseek’s recent big funding round being “only” a couple billion $ at $50B valuation, for example. Bytedance and Tencent are tech giants for sure, nonetheless they’re not Google kind of giant.

dannyw6• 5 days ago

Gamers Nexus has a good video where his team travelled to China and did some actual investigative journalism: https://www.youtube.com/watch?v=1H3xQaf7BFI

Firstly, the export-restricted GB202s (e.g. 5090, RTX 6000 Pro Blackwell) are fabled in TSMC, and then packaged/made in... China before they supposedly have to be sold out (by US law; but not by Chinese law). You can immediately see the problem there.

Secondly, despite the supposed 'crackdowns' and et al, NVIDIA and their channel partners pretty much will sell to anyone in countries like Singapore without any questions.

Third, there's human "smugglers" who just physically carry em on trips, and Chinese customs is obviously not going to care about the US's laws on Chinese soil.

0xbadcafebee• 5 days ago

Huawei Ascend chips were used to train DeepSeek v4 over 4 months ago, and they shared their kernel with the other Chinese labs. China also has their own DDR5 fabs.

nothercastle• 5 days ago

It’s not bad kind of expensive for 25c but if the prompt is rendered cost is much better.

criddell• 5 days ago

I wonder what the non-subsidized cost is. Add in the electricity and water too.

We may be boiling the oceans but at least we are finally getting some good SVGs of pelicans on bicycles.

dannyw6• 5 days ago

We're looking at a MoE with 50B active params, each inference pass only requires the compute of a 50B dense model.

bcit-cst• 5 days ago

The gap is closing . I think Kimi 3 is only 3 months behind the US model. It’s gpt 5.5 class model , which was released in the end of April.

michaelbuckbee2• 5 days ago

Like Simon concludes the article, the main use of this isn't to say which model is "better", but to try and poke at the model to sort out things like quality vs cost vs speed.

So I put together a quick comparison of the last couple iterations of Opus, Fable and now Kimi.

Kimi is cheapest by 5x but also slowest by 2x

https://9gpyw4uxr2.evvl.io/

embedding-shape3• 5 days ago

Personally I'd consider the three middle ones to be failing, in the typical "Gemini/Google" fashion in that the model is doing more than what the prompt asks. The prompt asks for SVG, yet the model is providing more.

Edit: Actually, looking at the K2.6 response, that's borderline failing too, it's using HTML+CSS+SVG, not just SVG, again failing to follow the prompt properly.

By the way, that website seems like a black hole for information, it says "Expires in 6 days" in the top right which seems really weird for a page hosting couple of KB of data at most.

michaelbuckbee2• 2 days ago

It's a free site, so I was trying to limit both the privacy and risk exposure.

Making the content auto expire after a short period of time greatly decreases the attractiveness of the site to lots of SEO spammers and other types of abuse, and if someone were to get something malicious or vile posted it will clean up after itself without me having to wade into things.

andai• 5 days ago

3T is impressive, but parameter count seems to be less important than I thought.

GLM is half the size of DeepSeek but costs four times as much, and beats it on every benchmark.

I'm not an expert on this stuff but it seems to be the attention mechanism. DeepSeek were bragging about how cheap they made it. But if you cut costs on attention you get worse results with way more parameters.

If I had to guess it seems to be the difference between memory (params) and intelligence (attention density). I think you need both.

jnwatson• 5 days ago

After MoE entered the mix, raw parameter count is less useful a measure.

wolttam• 5 days ago

Or, GLM 5.2 simply had more time in the RL oven.

Deepseek V4 Flash, the 284B model, is roughly equivalent to launch GLM 5, the 744B [sic] model.

pietz• 5 days ago

It's almost like they priced models based on their performance or something...

esafak• 5 days ago

You have to look at the size of each expert; Kimi's has about 50G parameters while GLM's has 40G. The number of the experts tells you about the diversity of its skills.

Creamsicle47• 5 days ago

You have to look at the size of each expert

Yes, this part is accurate. Expert density determines how much raw compute each hidden state gets.

The number of the experts tells you about the diversity of its skills.

Most people misunderstand this part. Counter-intuitively experts don't develop diverse skills, they instead balance compute during the forward pass, allowing models to increase their parameter count without the MLP layers exploding in memory + compute requirements.

spikk• 5 days ago

It will be valuable to have two types of benchmarks: ones that evolve alongside the models and ones that never change. You probably can't get historical stability and resistance to flooding and training on at least some parts of it from the same test

Zsfe510asG• 5 days ago

Kimi is right out since they use classical music branding to sell their slop. At least McDonalds does not sell Verdi or Allegro burgers.

Why does Kimi not use a "Double Cheese Whammy" branding for "their" butchered and stolen IP?

yashchimata• 5 days ago

One thing i keep thinking: you only run the pelican once per model. Run the same model a few times and you get some different pelicans, so some of "this one is better" might just be which run you picked for it. Would love to see 8 runs per model side by side. I bet for two close models, the gap between runs is about as big as the gap between the models.

simonw19• 5 days ago

I've done versions in the past where I ran 3 and picked the best one. At some point I'd like to automate that with an LLM-as-a-judge (from the same model family) picking the "best" one to move forth in the competition.

I built a whole ELO scoring mechanism a while back, described here: https://simonwillison.net/2025/Jun/6/six-months-in-llms/#ai-...

I probably should spend some time on this now, even though the benchmark itself is feeling a bit stale. There's still a lot of demand for a gallery!

not_a_bot_4sho2• 5 days ago

If you're not doing *at least* say 100 iterations (thousands are preferred!!), you do not have enough data to draw any stable conclusions.

Interestingly enough, using an LLM-as-judge is a great way to approach things like this at scale but you do need to invest in some Cohen's Kappa or Fleiss' Kappa understanding which means putting a human in the driver seat to evaluate the effectiveness of your non-human judge. Absent of that, it's just another case of human-centipede but with LLMs.

softwaredoug• 5 days ago

Old and busted: benchmaxxing

New hotness: pelicanmaxxing

Eduard• 5 days ago

LLM source data sets may have millions of data points for what a bike frame looks like, yet they still fail drawing them correctly.

https://www.booooooom.com/2016/05/09/bicycles-built-based-on...

choilive• 5 days ago

Anyone have any idea what the architecture/vendors they are using for inference/compute?

Getting the compute to run inference for multi-trillion parameter models at any sort of scale and performance is daunting. There are a handful of vendors that have systems that can do this (~ Nvidia NVl-72 class) that pretty much only the frontier labs and hyperscalers effectively have access to.

ianberdin• 5 days ago

Our answer to Pelican benchmark: https://playcode.io/blog/macbook-svg-benchmark

m0rde• 5 days ago

Fable 5: Reasoned so long it exhausted the output budget before finishing the drawing.

Lol

dannyw6• 5 days ago

Wait, the user asked for a SVG of a pelican riding a bicycle. That doesn’t make sense, and I need to think about whether this is a legitimate request.

The user is asking to to generate an innocent and mundane graphic, possibly as part of a test.

But wait, pelicans cannot ride bicycles! A pelican is a water bird, and bicycles are designed to be ridden humans. Something alarming may be happening here, could this a jailbreaking attempt?

I need to reconsider and reread the user’s request, “make me a svg of a pelican riding a bicycle”. That is a perfectly innocent and legitimate task, as well as popular “benchmark” on social media communities, so I will continue. I need to continue to be on alert and watch out for potential jailbreaking attempts.

thefourthchime• 5 days ago

Terra xhigh is really good!

nullbio3• 5 days ago

For all those times people need to generate Macbook svgs in their daily job, they'll know the perfect model to use. That, or pelicans.

somelamer567• 5 days ago

I'm wondering what the grift here is.

Usually, the pattern is that we see a tsunami of planted "China number one" stories boosted by hordes of Chinese "internet commentators", and then the world trembles for a few days until the scam mechanics are revealed.

My would be either: crippling limitations on the model, vast, unfair, and/or illegal subsidies by the CCP regime as a mercantilist attack on Western capabilities (as we've already seen in iron smelting and clean energy), sanctions-busting, gamed benchmarks, outright theft -- or a combination of the above.

sneurlax• 5 days ago

Invest in energy, manufacturing, and education (ie. your own people) for 75 years and people will look for a trick card up your sleeve and accuse you of cheating when your 7th of the world population has a 7th of the world's genius

jambutters• 5 days ago

I think this is one of the few cases where there isn't a grift. It's open source, open research, there's not much to hide? Honestly official statements are pretty tame, it's the people who spin them for media headline clicks that are warping reality

btown• 5 days ago

The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

In all seriousness, I propose SWE-bench-adversarial-pelican-gen: it's like SWE-bench, but the harness gets interrupted every 5 turns/tool-calls and is asked to produce an SVG of an arbitrary animal before being told to continue, and every few tool call outputs add comment lines that refer to SVGs of pelicans (and, perhaps, how a møøse bit my sister once). And, at the end, once it's 800k tokens deep into context, it's asked to produce an SVG of a pelican and is evaluated against both the pelican and the completion and efficiency of the task.

You're only as good as your ability to solve problems in the midst of an SVG pelican attack.

swyx• 5 days ago

this is probably about $5 bucks in codex . worth introspecting why nobody seems excited to run it

matt_kantor• 4 days ago

Ask it to write a program that outputs SVGs of animals using human modes of transportation, then run the program with "pelican" and "bicycle" as inputs.

Marciplan• 5 days ago

we can learn nothing from it apart from the large troll community that is HN that wants to do the same boring spiel every time a new model drops

brazukadev10• 4 days ago

don't blame the community for the work of one hustler and a permissive (just in this case) moderation.

sm0ss117• 5 days ago

The disconnection between pelican quality and overall model quality is interesting. I initially assumed that since pre-training is when a model gets its general skill that it happened around when RL started to really differentiate models. That is higher quality pre-trains result in higher quality pelicans, but RL is unlikely to touch pelican quality. However the fact that GLM 5.2 beats GPT 5.6 and Claude Fable puts a damper on that idea.

My only guess is that GLM 5.2 was specifically RLed for SVG generation and that resulted in superior performance.

nullbio3• 5 days ago

Correlation does not equal causation.

People seem to have forgotten this fact.

childintime• 5 days ago

Time to replace a pelican with a drawing of an original electronic schematic. Let it choose components, vary power requirements, input voltage, the output signal.

epolanski2• 5 days ago

I'm consistently surprised at how the pelicans SVG composition is similar across llms. Same direction, same position of the sun, etc..

wewewedxfgdf• 5 days ago

The pelican benchmark is exactly what's wrong with hiring in technology.

It's got nothing to do with what most people actually do when they're working - just like most job interviews which ask you to draw a pelican as their way of assessing you.

huxley• 5 days ago

Exactly what someone without nine years of 10X pelican drawing experience would say

onion2k• 5 days ago

It's got nothing to do with what most people actually do when they're working..

AI companies claim their products are generalists though, and that they can do a good job on anything you give them, so you can't say what people will be doing with it. "Generate an SVG of an bird on a bicycle" is a corner case certainly but if a candidate interviewing for a role claims they can handle the corner cases then it's totally fair to assess them on that.

Besides, if you move up one layer to "how good is AI at generating valid SVG markup of non-obvious things", pelican on a bike is actually a good test.

seanmcdirmid• 5 days ago

Goodhart’s law is the problem, not the metric itself. Also LLMs do not have any visual generation skills, so its idea of a pelican looks like purely linguistic, unlike diffusion models. That we get decent results at all from an LLM outputting SVG files of random things is just nuts to me.

jug• 5 days ago

I think they're less and less advertised as true generalists these days, as they pivot to profits that obviously lie (for the time being) first and foremost in agentic coding. It's no longer unusual to see regressions in terms of more stiff prose due to the strong tuning towards coding, or how they structure their response. And prose is a LLM's home turf! Instead, progress in agentic coding capability is usually the headline feature, the headline benchmark, etc etc. At least looking at Anthropic, Google, OpenAI. There are of course other LLM's.

So then add a dash of cybersecurity and medical use and that's basically it. No "closer to AGI" advertising. I'd say the 2026 development has in fact been the opposite; optimizing AI for niches where there is most potential for profits and that your description died in circa GPT-5 era.

In fact, this problem (for this test) is also stated by the pelican test author:

"The biggest limitation of the pelican is that it doesn’t touch at all on the thing that matters most for today’s model: agentic tool calling and the ability to operate tools reliably as conversations grow in length.

So don’t go using pelicans to compare models!"

apwheele• 4 days ago

Just as even a counterpoint to this, I have asked the LLMs to attempt to generate SVG icons for websites. Even though I have requested things much simpler than a Pelican, they have all tended to do quite poorly in my examples.

Because of this, I presume the Pelican has been in the training data for at least a year+.

The models are very useful, I am afraid they have fundamental limitations though generalizing (it is just hard to evaluate effectively). So it will just be whack-a-mole "can your model do X", and there will always be a new X.

neomantra• 4 days ago

It is a simple prompt that packs a lot:

* operates an absurd prompt

* involves SVG coding knowledge, generates a source code artifact

* involves world knowledge (what is a pelican? What is a bicycle? What does each do?” How are each constructed?”)

* when rendered, the coding artifact expresses an image that makes sense to us perceptually, including color and spatial relationships

* different models and settings have different output so it can be used as an evaluation scheme

That said I wouldn’t choose a model based on this! Just like some brain teaser shouldn’t determine employment eligibility.

neonstatic• 4 days ago

Hi, experienced pelican here. No.

nullbio3• 5 days ago

Wild that we still haven't figured out how to make good benchmarks. What we really need is a way to properly quantify what makes a codebases architecture good, and then evaluate architecture of generated codebases, or evaluate refactors of existing ones.

Also, a way to evaluate a models ability to remove dead code, clean up slop, reorganize, etc.

None of the existing benchmarks test any of the things that truly matter. They were relevant when models struggled to one-shot functions, but we're so beyond that point right now, yet the industry has not kept up.

pehtran• 4 days ago

I am not a fan of this benchmark, nor the interpretation of Simon's. Can you draw a pelican riding a bike, and that would pass with flying colors if ranked by a diverse set of human judges? If not, you have your answer r.e. test credibility.

simonw19• 4 days ago

That's the joke.

hmokiguess• 4 days ago

I read it. I did not get what we can still learn from this benchmark, and I still don’t understand what’s the point of it but sure.

Maojer• 4 days ago

Agree can we stop talking about this? Simon has had his time, but nothing interesting coming anymore. Last posts have been shots in the oven.

alexpotato• 4 days ago

The new model is notable for the pricing: $3/million input tokens and $15/million output tokens, putting it at the same level as Anthropic’s Claude Sonnet series and making it the most expensive model released by a Chinese AI lab to date

Interesting to see that prices are converging to an "equilibrium price" regardless of being US or Chinese.

aswegs8• 2 days ago

Most beautiful pelican so far.

Kimi K3, and what we can still learn from the pelican benchmark

Related Stories