Ask HN: Do coding assistants "see" attached images?

https://news.ycombinator.com/item?id=45488666

by cryptography • 10 months ago

3 12 10 months ago

news.ycombinator.com

I've been using Cursor and I'm genuinely curious about something.

When you paste a screenshot of a broken UI and it immediately spots the misaligned div or padding issue—is it actually doing visual analysis, or just pattern-matching against common UI bugs from training data?

The speed feels almost too fast for real vision processing. And it seems to understand spatial relationships and layout in a way that feels different from just describing an image.

Are these tools using standard vision models or is there preprocessing? How much comes from the image vs. surrounding code context?

Anyone know the technical details of what's actually happening under the hood?

yawpitch3• 10 months ago

They’re closed source black boxes, not even the people who built them really know what’s happening under the hood.

That said, one can reasonably infer that an LLM-based system isn’t doing any form of visual processing at all… it’s just looking at your HTML and CSS and flagging where it diverts from the statistical mean of all such structures in the training data (modulo some stochastic wandering and that it may, somehow, have mixed some measure of Rick Astley or Goatse into its multidimensional lookup table).

chistev2• 10 months ago

They’re closed source black boxes, not even the people who built them really know what’s happening under the hood.

Please explain

iswapna_2• 10 months ago

LLMs are trained to predict a bunch of tokens (GenAI produces text) from all the previous seen tokens, based on the data it was trained on. It does not understand anything about the spatial relationships like lines, objects etc in an image. "Not even the people who built them" - We have no real understanding of how LLMs work, yet. Traditional ML theory (classification/regression/clustering) largely does not apply to LLM's emergent capabilities like coding, arithmetic and reasoning. No such theory exists today. People are trying.

yawpitch3• 10 months ago

Imagine an equation several miles long, comprised of billions of variables, billions of constants, and billions of exponents.

You know none of the variables. None of the constants. None of the exponents. No one does, really, but even if you did it wouldn’t help, because no one bothered to write down the operators and the parentheses are randomly shifted around every time the equation is resolved.

All you know is that if you ask it for tea, it will always, invariably, and forever, give you back something that is almost, but not quite entirely, unlike tea. Sometimes it might be more unlike coffee, some times more unlike vodka and cow urine.

What you’ll never, ever, ever reliably know is what’s in the cup.

That’s about the best way I know to explain black box abstractions. In a few decades we might have a workable theory as to why these things function, to the degree that they do, though I’ll bet a rather large amount of money that we won’t.

gdsys2• 10 months ago

Have you tried asking one?

Claude gives what seems to be a reasonable answer.

https://www.perplexity.ai/search/how-do-ai-coding-assistants...

AznHisoka• 10 months ago

Thats not Claude…

pmxi• 10 months ago

You can select models on Perplexity. He used Sonnet 4.5 Thinking

gdsys2• 9 months ago

It is Claude. Via Perplexity.

Ask HN: Do coding assistants "see" attached images?

Related Stories