Pelicans on a bicycle

https://simonwillison.net/2024/Oct/25/pelicans-on-a-bicycle/

by colejohnson66 • 2 years ago

102 54 2 years ago

flippyhead• 2 years ago

Once again, Claude wins.

twiss• 2 years ago

I think it's interesting to consider how humans would go about this task ("Generate an SVG of a pelican riding a bicycle"), and how well they would do if they had to output the SVG into a text box without any other tools. Considering that, I think Claude 3.5 Sonnet and GPT-4o did incredibly well, and even the others might be commended for making a valid SVG at all..

nemomarx• 2 years ago

depends on the human, right? I imagine an artist who specializes in SVG would do pretty well and might make it a professional logo?

egypturnash• 2 years ago

Pro artist who specializes in vector work here: I would use Adobe Illustrator to draw it, while looking at actual photos of pelicans and bicycles, and export an SVG. If it needed to have a lot of named parts I could make that happen.

If I had the latest version of Illustrator then I would consider seeing how well its image generation does, but I do not because it has a lot of exciting new bugs that break my normal workflow. I believe that under the hood that works by feeding your text prompt to a bitmap image generator and running the same old autotrace on it, which results in some pretty messy and hard-to-edit shapes.

danielcorin• 2 years ago

More recently, `gemini-exp-1206` did quite well [1].

[1] https://github.com/simonw/pelican-bicycle/blob/main/README.m...

toxik2• 2 years ago

I feel like ”quite well” is overselling it a bit. It did maybe better.

ttul• 2 years ago

Gemini 1206 is the new hotness in my books. I've moved my day to day LLM needs over to Google's tab for the first time. I'm not sure what they changed, but it deserves a good look. Claude 3.5-Sonnet (New) is fantastic as well, but the 2M token context window offered by Google allows you to suck in an entire code repository and reason effectively across the whole thing. Google is catching up...

simonw4• 2 years ago

I've been using this dumb benchmark for a few months now. More posts about it here: https://simonwillison.net/tags/pelican-riding-a-bicycle/

behnamoh• 2 years ago

[flagged]

simonw4• 2 years ago

Sure, they're the wrong tool for drawing a pelican - but testing their SVG output is a useful way to get a feel for how good they are at step by step reasoning, coordinate systems, spatial awareness and generating valid SVG/XML.

There are genuinely useful applications of SVG-generation from LLMs - outputting simple infographics or charts for example.

I use LLMs to write HTML all the time, of which SVG is a useful optional component.

gffrd• 2 years ago

these models aren't pelican painters or anything like that, they're LANGUAGE models

Tools are defined by what people use them for, not by how they were intended—or designed—to be used. (Just ask Nvidia)

adding: so I think someone comparing how various tools perform at a task that's valuable to them—and probably others—is just fine, even if it's different from what the creator of the tool intended?

sfink3• 2 years ago

I hope you have other private benchmarks running that you don't talk about or publish, just in case a model maker intentionally targets one of your benchmarks, or some fuzzy "find things people have mentioned as potential LLM benchmarks" process scoops up your ideas and/or outputs.

zamadatix2• 2 years ago

If we ever get to the point LLMs are already optimized to answer every question you can think of then there isn't really a need to have a secret question in the first place.

NotSammyHagar• 2 years ago

I saw your recent post on running Llama 3.3 70B on a m2 pro 64 gb. Do the many variants of apple silicon alternatives with varying numbers of cpus, gpus, and neural engines matter that much for how fast these llms can generate tokens, answer questions? More hw is always better, but what can we say how performance scales with the many different choices?

64gb ram is crucial, after that, need 1+ tb storage, and then?

simonw4• 2 years ago

I don't know. I believe memory bandwidth matters, and I got the impression that the M4 series isn't yet as good as the M2 was on that front, but I'm half-remembering things I've heard here.

dowager_dan99• 2 years ago

Weak - the pelican is not pedaling in any of these!

eminence32• 2 years ago

I like how one of them is clearly (to my eyes, at least) a person holding a gun

MarkusWandel3• 2 years ago

I'm focusing on the bike part here because, as a bike geek, I could draw one from memory that's correct in all details. But to a non-bikie that's more difficult than you'd think. I can't find the picture gallery right now but an article about it, which links another article:

https://web.archive.org/web/20240419001426/https://www.wired...

So the fact that the AI models screw this up so badly is understandable. Sure, they screw up in ways that humans wouldn't, such as the beak backwards in one of the pictures (pointy end toward the bird!) because they don't know or care about something every human would know: What a beak is for and what it looks like in general. Or for that matter the biodynamics of how a pelican's long, spindly legs could, in fact, work a pair pedals. But ask me to draw a pelican from memory, and have a good laugh (if you're better at it than me) because to me, they're just kind of a peripheral vision, pink abstraction, not something I focus on understanding. And that's what they are to the AI model too.

parpfish• 2 years ago

... they're just kind of a peripheral vision, pink abstraction, not something I focus on understanding

are there pink pelicans, or are you thinking of flamingoes?

MarkusWandel3• 2 years ago

Ha, see, even missed that part! Honestly.

larubbio• 2 years ago

This is the the artist you are thinking of.

https://www.gianlucagimini.it/portfolio-item/velocipedia/

alwa2• 2 years ago

This is incredible. I wonder if anybody has set out to build some of these bikes as sculptures.

Ylpertnodi• 2 years ago

as a bike geek, I could draw one from memory that's correct in all details.

Mo link, sorry, but on youtube, GCN asked pro riders to draw a bicycle...none could.

MarkusWandel3• 2 years ago

Well, as a bike geek - who wrenches on them, changes old bikes to different configurations etc - I can visualize every part because I've dealt with all of them. I can tell you, for example, that ancient Shimano downtube shifters are held in place by an M4.5 bolt. M4.5? Try to find something to fit that at your local hardware store (when changing said bike to handlebar-mounted shifters). Or which way the opposite sides of a BSA bottom bracket are threaded (from memory! Which side has the backwards threads?) Or the whole stack of bits and pieces that make up a headset (both threaded and threadless).

Whereas a pro rider can probably tell you all about the biomechanics of how to optimally interact with the bike, the right foods to eat and how much to sleep and when. But the actual wrenching around with them? That's the pro mechanic's job.

vbarrielle• 2 years ago

Lots of pro riders do not take care of their bikes themselves. They're used to having bike mechanics adjust everything for every race, and a lot of them don't consider it part of the job to take care of their training bikes. Some of them don't even do the cleaning.

eitally• 2 years ago

Those output examples are absolutely horrifically bad compared to what I get with a cursory request to Gemini 2.0 using Imagen3 via gemini.google.com.

toxik2• 2 years ago

This is generating SVG data, not using an image generator.

alwa2• 2 years ago

Was yours an SVG? I think that’s what makes Simon feel that this test is useful: the LLM has to generate functioning SVG code describing these shapes.

simonw4• 2 years ago

Yeah, this test is to see if a pure LLM can output SVG that renders well. It's effectively a test of their "spatial reasoning" capabilities.

oatsandsugar• 2 years ago

It looks like no LLM has seen a pelican before at all.

pjs_• 2 years ago

I wanna see this running in a feedback loop - show the model its output and get it to make corrections.

Remember that these are basically one-shot. Very different to how you or I would solve the problem (get a circle up on the screen, have a look at it, make some changes, add some wings, tweak the dimensions, etc.). We would go through hundreds or thousands of feedback cycles before we got something half-decent -- in this situation the model only gets one attempt.

cadamsau• 2 years ago

Careful not to shout too loud about it or they’ll start training for it!

Latest Claude does a suspiciously good job…

marcodiego• 2 years ago

Most humans can't correctly draw a bicycle from memory: https://www.wired.com/2016/04/can-draw-bikes-memory-definite...

fuegoio• 2 years ago

Mistral Large proposition: https://chat.mistral.ai/chat/4da427b1-e033-454d-b134-c5d1f6e...

Mistletoe• 2 years ago

If only it didn't have to be svg, so glorious.

https://g.co/gemini/share/56e5dfa1a598

skissane2• 2 years ago

Just thinking out loud: a lot of models generate pretty poor SVG output, but part of that is because they don't get any visual feedback.

But, what about this workflow: given prompt, LLM generates two SVG outputs. Both are rendered by an SVG renderer, and then we combine the two into one image, one on the left and the other on the right. We then ask a visual LLM (could be the same LLM or could be a different one) to tell us whether the left half or right half of the image is a better response to the prompt. Now we've got preferences which can be used to fine-tune the LLM using DPO. And you could iteratively repeat the process – as the LLM is fine-tuned it may produce even better outputs which then produces new preferences for further fine-tuning.

Would be interesting to see what kinds of results it might produce in practice.

codetrotter2• 2 years ago

Would be interesting to see what kinds of results it might produce in practice.

I would not be surprised to see that the LLM generating SVG does a UNO reverse, and makes use of little colored squares in a grid to draw a “vector image” where each of the squares represents individual pixels :p

skissane2• 2 years ago

Such a “vector image” would be very large, if you put a limit on the length of the generated SVG the LLM would be blocked from pursuing that strategy. Even without an explicit limit, the LLM’s context limit may block it: if the resulting SVG is larger than the LLM’s context, the LLM may forget what it is doing part way through generating the SVG, which is unlikely to produce a good result.

e3a8• 2 years ago

"Generate room with no elephant" https://imgur.com/BI61S1T

Zopieux• 2 years ago

Models hate negative prompts. There's a reason Stable Diffusion txt2img requires a dedicated "negative" CLIP input in addition to positive.

FLUX, for all its benefits, is really annoying in this aspect as you cannot really provide negative guidance without complicated hacks [1]. Therefore, "orignal" ideas that go against popular wisdom or mainstream culture will be very hard to prompt for. This is, in my opinion, an important caveat of ML "art". ML "art" is already considered useless slop because of how much it conforms to biases built in the weights. The inability to tune down adherence to mainstreaming norms is therefore all the more problematic.

[1] https://www.reddit.com/r/StableDiffusion/comments/1estj69/

boredhedgehog• 2 years ago

Some of the drawings, like the one from Amazon Nova Pro, are quite fascinating as abstract artworks. It's like the idea of a bicycle without its physicality.

tessellated• 2 years ago

aren't any pelican on a bicycle SVG files floating around (yet)

maintains website collecting SVG files of pelicans on bicycles

kayo_20211030• 2 years ago

I searched the comments and nobody's mentioned that none of the bicycle riders looked like pelicans. Seriously, a Pelican looks a certain way, ... more than his belly can and all that.

yowzadave• 2 years ago

Some friends noticed another one the other day—none of the image generators were able to generate a “glass of wine, full to the brim.” All of the images were just of regular partially-filled glasses of wine, with the accompanying text trying to assure you that the glass was indeed full to the brim.

Pelicans on a bicycle

Related Stories