How far are we from intelligent visual deductive reasoning?

by belter • 2 years ago

124 118 2 years ago

minimaxir2• 2 years ago

GPT-4-Vision is a surprisingly lot better than I expected for a variety of practical tasks. OpenAI released a demo notebook with practical use cases for tagging/captioning a few weeks ago: https://cookbook.openai.com/examples/tag_caption_images_with...

It will be much better once it gets function calling/structured data support so you don't have to the formatting hacks that notebook abuses to get a specific output format.

Just don't ask it to describe a real person in the image or it gets fussy.

WhitneyLand2• 2 years ago

Gpt-4 will describe people, just don’t ask it to be judgmental.

Prompt:

Describe all objective details you can glean from the person in this photo.

Result:

The person in the photo appears to be a woman with fair skin. She has long, straight, light blonde hair, and blue eyes. Her eyebrows are well-shaped and lighter in color, matching her hair. She is wearing mascara and possibly eyeliner, which highlight her eyes, and she has a natural, possibly light pink, lipstick on. She is smiling, showing her teeth, which are white and well-aligned.

The woman is wearing what seems to be a black outfit, although only a small portion is visible. She appears to be indoors, with a nondescript background.

There are no visible tattoos, piercings (other than possibly in her ears, which are not fully visible), or distinctive marks that can be seen in the photograph.

sandspar• 2 years ago

ChatGPT is happy to describe people if it has the option to give a flattering description. Our current chatbots' basic relation to humanity is flattery.

carlossouza• 2 years ago

We observed that all the VLMs tend to be confident while being wrong. Interestingly, we observed that even when the entropy was high, models tried to provide a nonsensical rational, instead of acknowledging their inability to perform the task

It looks like all current models suffer from an incurable case of Dunning–Kruger effect cognitive bias.

All are at the peak of Mount Stupid.

leereeves• 2 years ago

How can a neural network evaluate "confidence"?

The parameters don't store any information about what inputs were seen in the training data (vs being interpolated) or how accurate the predictions were for those specific inputs.

And even if they did, the training data was usually gathered voraciously, without much preference for quality reasoning.

habitue2• 2 years ago

I don't know for sure, but here's a plausible mechanism for how:

Multiple sub-networks detect the same pattern in different ways, and confidence is the percent of those sub-networks that fire for a particular instance.

There's a ton of overlap and redundancy with so many weights, so there are lots of ways this could work

montjoy• 2 years ago

IIRC confidence in video is related to predicting what happens next vs what actually happens. If the two seem to correlate to the model it would give it a higher confidence ranking, which would then be used further for self-reinforced learning.

pbhjpbhj3• 2 years ago

Phind gives me ChatGPT answers with relatively authoritative references to works on the web that (usually!) support the answer. Could it have a post-filter to fact check against the references?

I guess that is a slight variation of the sibling (@habitue's) answer; both are checks against external material.

I wonder if best resources could be catalogued as the corpus is processed, giving a document vector space to select resources for such 'sense' checking.

XenophileJKO• 2 years ago

Actually you can get a very good proxy by looking at the probability distrobution of the "answer" tokens. The key here is you have to be able to identify the "answer" tokens.

https://arxiv.org/abs/2402.10200

nyrikki2• 2 years ago

LLMs are trained to sound confident.

But they can also only do negation through exhaustion, known unknowns, future unknowns, etc...

That is the pain of the Entscheidungsproblem.

Even in Presburger arithmetic, Natural numbers will addition and equality, which is decidable, still has a double factorial time complexity to prove. That is worse than factorial time for those who've not dealt with it.

Add in multiplication then you are undecidable.

Even if you decided to use the dag like structure of transformers, causality is very very hard.

https://arxiv.org/abs/1412.3076

LLMs only have cheap access to their model probables which aren't ground truth.

So while asking for a pizza recipe could be called out as a potential joke if add a topping that wasn't in its training set, through exhaustion, It can't know when it is wrong in the general case.

That was an intentional choice with statistical learning and why it was called PAC (probably approximately correct) learning.

That was actually a cause of a great rift with the Symbolic camp in the past.

PAC learning is practically computable in far more cases and even the people who work in automated theorem proving don't try to prove no-instances in the general case.

There are lots of useful things we can do in BPP (bounded probabilistically polynomial time) and with random walks.

But unless there are major advancements in math and logic, transformers will have limits.

astrange• 2 years ago

That's not how Dunning-Kruger works. There's never a point where incorrect people were more confident than correct people.

https://en.m.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effec...

CamperBob26• 2 years ago

When self-driving cars were first becoming a thing, I was one of the skeptics who would say things like, "Call me when the AI can avoid situations like https://i.imgur.com/cmR2PDm.jpeg ." At the time, my thinking was that the software would have to recognize both fixed and moving objects nearby, capture their position and velocity, infer or look up their inertia tensors, and run at least a basic dynamics simulation to anticipate and prevent accidents like the one that's about to happen in the photo.

However, I didn't consider the possibility that such a simulation might emerge from a model driven by nothing but visual input, given a large-enough data set and the right training. At this point my old argument seems like a losing one, even if present-day systems can't answer the "What happens next?" question reliably enough to trust in all driving situations. It won't exactly have to be perfect in order to outperform most humans.

As a matter of fact, I just checked, and the situation in the meme is already partially recognized by ChatGPT 4: https://i.imgur.com/wLSBSkJ.png , even if it misses the comedic implications of the truck heading for the overpass. Whether it was somehow trained to give a useful, actionable answer for this particular question, I don't know, but the writing's on the proverbial wall.

-------

Edit due to rate-limiting: note that I intentionally cropped out the WAIT FOR IT frame when I submitted the pic, so that it wouldn't see it as a meme loaded with comedy value. When I explicitly asked it what was funny about the image, ChatGPT4 eventually answered:

"When the truck hits the overpass, given that the convertible is following closely behind, the occupants could be in for quite a shock. The impact might cause the portable toilets to become dislodged or damaged, potentially resulting in the contents being spilled. For the people in the convertible, this could mean an unexpected and unpleasant shower, especially because the convertible's top is down. The humor perceived here is in the anticipation of the unexpected and the sudden reversal of fortune for the individuals in what appears to be a fancy car – it's a form of slapstick humor."

But I did have to provide a couple of strong hints, including telling it that the truck was not towing the car ( https://i.imgur.com/AlPRgEQ.png ). I was going to feed Claude the same hints, but it repeatedly failed to recognize the SMS code for registration.

minimaxir2• 2 years ago

Video data doesn't work well with these multimodal LLMs because images are still somewhat intensive for them process. Both the GPT-4-Vision demos and Gemini take a few frames from each video (~1 frame per second).

Self-driving cars use much more efficient approaches for the real-time outputs needed.

CamperBob26• 2 years ago

Sure, but my point is that self-driving may be easier to achieve if we don't insist on doing it the hard way with hardwired logic running task-specific models, the way I originally assumed it would have to be done.

habitue2• 2 years ago

Not to detract from your point, because I think you're right, but gpt4 doesn't seem to mention the biggest thing a human wpuld notice, which is that the portable toilets are about to be knocked off the truck by driving under the bridge.

still, impressive

sfn42• 2 years ago

How do we know the cars are moving? It doesn't seem clear to me, they could both be standing still.

dmd4• 2 years ago

Claude 3 Opus does a little better https://shot.3e.org/ss-20240310_141855.png but still doesn't get the interaction with the convertible.

ickelbawd• 2 years ago

Yeah, not sure why it missed it. But when prompted with the photo and "What's funny about this photo?" GPT-4 replied:

"The humor in this photo stems from the anticipation of an unfortunate event that seems likely to occur. A portable toilet is being transported on the back of a truck that appears to be entering a tunnel with a height limit, and a convertible car with its top down is closely following the truck. The implication is that if the portable toilet were to hit the tunnel entrance due to being too tall, its contents could potentially spill into the open convertible behind it, leading to a very messy and unpleasant situation. The text "WAIT FOR IT" plays on the tension of expecting something bad to happen, adding to the comedic effect."

renegade-otter2• 2 years ago

Now imagine an automated system deciding in a checkmate situation whether to have a bigger collision or a smaller collision but risking to be covered in human waste.

By the way, that output is pretty freaky. I just can't imagine the amount of data needed to get to that level of accuracy.

pbhjpbhj3• 2 years ago

In the Netflix series "Upload" (about minds living an after life in shared computer simulations; spoilers!) the real-World has fully autonomous vehicles, one feature is "passenger protect", whether you bias the vehicle to protect the passengers or people outside the vehicle. Within the show it's considered immoral for people to protect themselves over pedestrians, but people choose that option anyway.

A central character dies in a car crash, and another character says something like 'you were murdered, you died in a car crash', the implication being cars are so safe the only way to die in one is to have someone murder you.

It touches on some interesting points (though overall it's a little vapid and soapy, imo).

klyrs3• 2 years ago

Meanwhile in reality, a truck got hit twice by automated vehicles because it was hitched to a tow truck and travelling in a direction the "ai" considered unrealistic. Proposed name change #4532: Accelerated Stupidity

levocardia2• 2 years ago

Also in reality, about 80 Americans will die today from being hit by a human-driven car. Ditto for tomorrow, and the next day, and the day after that.

cgearhart2• 2 years ago

The Georgia Tech Knowledge Based AI course involved building a program to answer ravens progressive matrix questions. The course was offered in the online MS program, so thousands of students have taken the course. The most impressive result I saw was one student who got nearly perfect results in about 25 lines of Python code.

This may be a case where humans do well on the test, but you can do very well on the test without doing anything the way a human would. The fact that GPTs aren’t very good at the test isn’t probably evidence that they’re not really very smart, but it doesn’t really mean that if we fix them to do very well on the test that they’ve gotten any smarter.

mjburgess18• 2 years ago

There are an infinite number of algorithms to compute A from Q, given a se tof (Q, A). Almost none, surely, are intelligent.

These proxy measure of intelligence are just arguments from ignorance, "I don't know how the machine computed A from Q, therefore...".

But of course some of us do know how the machine did it; we can quite easily describe the algorithm. It just turns out no one wants to because its really dumb.

Esp. if the alg is, as in all ML, "start with billions/trillions of data points in the (Q, A) space; generate a compressed representation ZipQA; and for novel Q' find decompressed A located close to Q similar to Q'"

There are no theories of intelligence which would label that intelligence.

And let me say, most such "theories" are ad-hoc PR that are rigged to make whatever the latest gizmo "intelligent".

Any plausible theory begins form the initial intuition, "intelligence is what you do when you dont know what you're doing".

andoando9• 2 years ago

I remember getting a lot of flak for saying a purely statistical framework is not going to achieve human level intelligence, but I still firmly believe that.

I also believe the path forward is research in knowledge representation, and even now when I search for it, I can barely find anything interesting happening in the field. ML has gotten so much interest and hype because it’s produced fast practical results, but I think it’s going to reach a standstill without something fundamentally new.

thfuran• 2 years ago

But does the fact that we don't really know how human intelligence works actually suggest that it is categorically smarter?

mewpmewp24• 2 years ago

What is more special about how human intelligence works? In the end we are all particles and it all could be trillions of data points very simplistically interacting with each other resulting in emergent behaviour and complex, intelligent results.

We know how common molecules can interact with each other. Does this mean that anything built on top of them is not intelligent?

Everything in life could just be "statistics".

Legend2440• 2 years ago

There are no theories of intelligence which would label that intelligence.

Actually, compression-is-intelligence has been a relatively popular theory for a couple decades. The Hutter prize from 2006 is based around that premise.

The idea is that compression and abstraction are fundamentally the same operation. If you had a perfect compressor that could compress the digits of pi into the 10-line algorithm that created them, in a deep sense it has understood pi and could generate as many more digits as you want.

epups• 2 years ago

How many libraries were included in these 25 lines of code?

samatman• 2 years ago

At least one, obviously, and a rather large one at that.

The point of the comment you replied to is that conventional CV software can recognize the patterns in tests like Ravens Progressive Matrices just fine, and a simple logic tree can then solve them, while the LLM approach is still struggling to get the same result.

This a commonplace shortcoming of the current generation of LLMs: ironically, they often fail at tasks which computers can do perfectly using conventional software.

cgearhart2• 2 years ago

Obviously “Python” was used…and then it used numpy because the image format in the assignment was numpy arrays. However, the 25 lines was basically “sum the rows and sum the columns then compare those vectors” or something like that. This wasn’t really a case of all the complexity being hidden in a dependency; it was a case of finding a very simple heuristic that made the problem trivial.

bee_rider• 2 years ago

Could a GPT write those 20 lines of code?

tflol• 2 years ago

maybe if those 20 lines of code are in a private github repo somewhere

AlienRobot• 2 years ago

What I find odd is the human obsession with making AI as smart as a human.

In the early periods of AI image generation, you would see AI generate images so uncanny no human could make something like that. It was genuinely unique and otherhumanly, worth of being called art.

We "fixed" that bug so we could make the world's most expensive face swap app.

bongodongobob• 2 years ago

Nonsense, you can still make weird/creative stuff. It seems like you're speaking about the marketing materials that are pushed out.

miguelrochefort• 2 years ago

CS7637 was a fun course! One can get pretty far (80% accuracy) by using xor masks and comparing centroids.

GaggiX2• 2 years ago

Nowadays VLMs are kind of Frankenstein models where the vision model is separated from the LLM and the communication is only one way where the vision embeddings are projected into textual embeddings and aligned; I think that this method severely limits the visual capability of the model and I think that it would be fixed by training a single decoder-only transformer on both textual and image tokens, I think that this method would allow some new incredible emergent behaviors like accurately translating images of logos to SVG, actually good ascii art, more meaningful openscad scripts and the ability to generate images with more meaningful composition and text (for now models can only generate meaningful text if it's available in the prompt, this is true even with Dalle 3 as the prompt is rewritten by ChatGPT).

famouswaffles• 2 years ago

For now, LLMs are a lot better when you express this kind of task in text, preferably as a 1D arrangement. GPT-4's performance on the Abstraction and Reasoning Corpus. https://arxiv.org/abs/2305.18354

Also, this model trained to predict image tokens seemed to be able to perform Raven's Matrices type questions when expressed as an inpainting task in context. https://yutongbai.com/lvm.html

andoando9• 2 years ago

Check out the ARC benchmark. https://lab42.global/arc/

Iirc, humans get something like 80% on the test, and the best performing AI models are at 25%

2snakes• 2 years ago

To answer the question: How far from visual deductive reasoning? Consider G. Spencer Brown's Calculus of Indications. If you read his book it shows how using the mark/cross is trivial for visual deductions (if well-formed in translation).

There is also work on this from Kaufmann at University of Illinois and William Bricken.

Though this is a bit different than Raven's matrices.

levocardia2• 2 years ago

A little surprised that neither the paper nor the discussion here cite Francois Chollet's 2019 paper on defining intelligence for machines [1], which talks explicitly about visual reasoning and proposed an RPM-like visual task as a benchmark. I found it very useful for thinking about what it means for an AI to be generally intelligent.

From what I remember (having not read it in a while), his idea is basically that general intelligence is the efficiency with which you can learn, after appropriately "penalizing" any baked-in priors or knowledge beyond what a typical animal is born with.

As an example, Magnus Carlsen's chess skill is a testament to his general intelligence because he became skilled at chess in one human lifetime, starting out with only basic priors about the world like objectness, agentness, and some others I can't remember. In contrast, DeepBlue is not as intelligent, because it started with many more priors specific to chess, and AlphaZero is also not as intelligent, because it acquired its chess skill much less efficiently, requiring the equivalent of thousands of human lifetimes.

I don't remember if the paper considers whether 3.5 billion years of "pre-training" counts in this regard, though.

[1] https://arxiv.org/abs/1911.01547

SmooL• 2 years ago

Isn't that a bit of a cop-out though? They're just redefining the word "intelligence" to be something else (here, the ability to learn new things). That's fine and all, but that doesn't answer the question we actually care about, which is the absolute magnitude of the ability. It doesn't matter how "long" an entity took to learn, or how "efficient" they were in learning - at the end of the day, AlphaZero will crush Magnus.

mynameisnoone• 2 years ago

Holy shit. I was thinking about this use-case the other day. I was thinking about whether a deep learning model could draw on the internet to identify objects in film screen captures and determine what they were or if they were unique.

For example, I was watching the news and wondered if a mostly submerged vehicle was either a modern Jeep or an older Land Rover. Certainly AI/ML is the right tool for this task.

How far are we from intelligent visual deductive reasoning?

Related Stories