IQ tests results for AI

https://www.trackingai.org/home

by stared • 11 months ago

220 343 11 months ago

amunozo• 11 months ago

Babe wake up. New benchmark to overfit models just dropped.

testdelacc1• 11 months ago

They’re definitely going to overfit on this, but this will be much better from a marketing perspective. Normies don’t know wtf an MMLU is, but they do know what IQ is and that 140 is a big number.

Can’t wait for CEOs to start saying “why would we hire a 120 IQ person who works 9-5 with a lunch break when we can hire a 170 IQ worker who works 24x7 for half the cost??”

codr7• 11 months ago

You won't have to, it's already happening.

notahacker2• 11 months ago

"Workers rejoice as model overfitted to score 170 on IQ test turns out to be incapable of performing basic tasks..."

perching_aix• 11 months ago

I sure hope they do, will finally give people some sense of just how little IQ tests actually say.

Which mirrors my general experience with automation and AI. Once something is automated and/or AI can do it, the magic goes away, and we're that much ahead in peeling back what is it that actually differentiates us and how exactly.

scotty792• 11 months ago

They have offline test that's supposedly not in the training data. It gets lower scores but best one is still 120 IQ.

dang• 11 months ago

Can you please make your substantive points thoughtfully and avoid the snarky one liners here, tempting though those of course are?

https://news.ycombinator.com/newsguidelines.html

pdhborges• 11 months ago

If the AI is so smart why are we feeding so many dumb humans?

jonplackett• 11 months ago

Really need to use a CDN before you get #1 on HN

kator• 11 months ago

LLM vibe coded site and architecture?

scotty792• 11 months ago

Dumping things on Cloudflare is clever architecture now?

mirekrusin3• 11 months ago

How many microservices, sql joins, distributed kafka piplelines etc. we currently recommend for serving static, public article?

diggan• 11 months ago

I mean not really, as always you just need to make sure you're not doing 10s of dynamic calls for each page load and if you do, add some minute-long cache at least. Most of the stuff that gets hugged to death really shouldn't, most of the times it's just static content that is trivial to host on even $10/month instances.

FranOntanaya• 11 months ago

The amount of calls on some pages displaying the simplest stuff is mind-boggling. 160 requests for a page just displaying a HTML5 video and a title, 360 requests for a Reddit page, it's nuts. We don't need to be like this.

stared2• 11 months ago

It was not my intention to bring the HN hug of death.

(For a reference, I shared a link, I am not the author.)

charles_f• 11 months ago

I'm on a shared hosting instance with relatively low resource allocation but reasonable bandwidth, and made #1 several times while never having issues loading. As long as your content is static and doesn't generate load on your server, you should be fine serving a lot of concurrent requests. Issues start when serving content relies on a database, or you serve large content

habibur• 11 months ago

caching is the solution. don't serve dynamic content w/o html caching.

ekianjo• 11 months ago

not if you have a static site

ranguna• 11 months ago

Iirc HN itself does not have a CDN.

gpt5• 11 months ago

The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.”

They then hypothesized a general factor, “g,” to explain this pattern. Early tests (e.g., Binet–Simon; later Stanford–Binet and Wechsler) sampled a wide range of tasks, and researchers used correlations and factor analysis to extract the common component, then norm it around 100 with a SD of 15 and call it IQ.

IQ tend to meaningfully predicts performance across some domains especially education and work, and shows high test–retest stability from late adolescence through adulthood. It is also tend to be consistent between high quality tests, despite a wide variety of testing methods.

It looks like this site just uses human rated public IQ tests. But it would have been more interesting if an IQ test was developed specifically for AI. I.e. a test that would aim to Factor out the strength of a model general cognitive ability across a wide variety of tasks. It is probably doable by doing principal component analysis on a large set of benchmarks available today.

krapp• 11 months ago

I imagine the value of something like this is for business owners to choose which LLMs they can replace their employees with, so it using human IQ tests is relevant.

azernik2• 11 months ago

The point is that the correlation between doing well on these tasks and doing well on other (directly useful) tasks is well established for humans, but not well established for LLMs.

If the employees' job is taking IQ tests, then this is a great measure for employers. Otherwise, it doesn't measure anything useful.

YetAnotherNick• 11 months ago

ARC-AGI challenge aims for that. In fact the objective is even more strict that the tasks must be trivial for most humans given time.

nsoonhui• 11 months ago

Another component of this theory concerning g is that it's largely genetic, and immune to "intervention" AKA stability as you mentioned. See the classic "The Bell Curve" for a full exposition.

Which makes me wonder what's the point of all the intervention in the form of teaching/parenting styles and whatnot, if g factor is nature and immutable by large? What's the logic of the educators here?

cedilla• 11 months ago

"The Bell Curve" is, let's say, highly controversial and not a good introduction into the topic. Its claim that genetics are the main predictor of IQ, which was very weakly supported at the time, has been completely and undeniably refuted by science in the thirty years since it's publication.

lukan• 11 months ago

Assuming the assumption it is true (which I doubt) - there obviously is still value in teaching knowledge, so making students know more and practical skills, not produce more intelligent students.

You can have a IQ of over 200, but if no one ever showed you how a computer works or gives you a manual, you still won't be productive with it.

But I very much believe, intelligence is improvable and also degradable, just ask some alcoholics for instance.

pbmonster• 11 months ago

Isn't this argument directly countered by the fact that you can study for IQ tests and subsequently do better?

depressedpanda• 11 months ago

If a child is, e.g., two standard deviations below the norm, it is cruel to expect it to keep up with the pace of other students.

Education can be better adapted to the child's needs.

jdietrich• 11 months ago

G is (largely) immutable, but knowledge and skills are not. The economy is not zero-sum and we all benefit from increasing the total amount of human capital. Unfortunately, thinking around education is dominated by people who wrongly believe that the economy is zero-sum.

matthewdgreen3• 11 months ago

If IQ potential was 50% genetic, then the teaching would potentially raise your actual IQ by affecting the other 50% which is huge. IQs scores in populations and individuals change based on education, nutrition, etc. But even if we hypothesized a pretend world where “g” was magically 100% genetic, this (imaginary) measure is just potential. It is not true that an uneducated, untrained person will be able to perform tasks at the level of an educated, trained person. Also The Bell Curve was written by a political operative to promote ideological views, and is full of foundational errors.

tim3334• 11 months ago

Education is to learn stuff, not increase your IQ.

terminalshort3• 11 months ago

There is no logic. Some people are just born much smarter than others and there is nothing you can do about it if you aren't one of them. The implications of that fact are too much for most people to accept so we pretend that it isn't true.

breakyerself7• 11 months ago

It's not genetic. The bell curve is garbage. It's a not a work of science. It's a work of political argumentation built on pretending to be science.

regularization• 11 months ago

concerning g is that it's largely genetic, and immune to "intervention" AKA stability

Biology seems to be the destination for smuggled in quasi-religious beliefs. Lysenkoism, creationism and in the case of a segment (or more?) of the western professional-managerial class, this kind of Bell Curveism.

In the past people were satisfied with the divine apotheosis of people into a superior Brahmin class, or chosen people, or even in more modern times a Calvinistic elect, which this is an attempt to smuggle in as a secular, "scuentific" basis.

If we were watching elementary particles smash together in an accelerator, the idea that a brain could be boiled down to a number and ranked in order, and said this be due to differential genes and such would be seen as absurd. Especially considering human behavioral modernity happen two to three thousand generations ago, if one knows a genetic time scale. For our version of biological Lysenkoism or creationism, this all goes out the window though. Speaking of Lysenkoism, it is akin to the Marxist idea of false consciousness - the people who believe such ideas can see the errors of Lysenkoism or creationism, but the crank idea tied to their particular system makes a lot of sense to them.

I think of al-Andalus in Spain the 1450s, or the Battle of Vienna in 1683. Until a few centuries ago, Europe could barely keep itself free of Arab or Turk rule (and often didn't). Change back a few centuries and this would be about the genetic superiority of the Arab brain over the Caucasian. It's all quite silly.

twodave• 11 months ago

One’s ceiling may be more or less stable, but there are many instances where individuals have certain underdeveloped cognitive skills (of which there are a litany), undergo training to develop those skills, and then afterwards go on to score (sometimes much) higher on IQ tests. Children with certain disabilities such at autism or FASD tend to see more dramatic differences. This isn’t to say they “became more intelligent,” but rather that the testing is unable to measure intelligence directly, rather relying on those certain cognitive skills as a proxy for intelligence.

csa2• 11 months ago

Which makes me wonder what's the point of all the intervention in the form of teaching/parenting styles and whatnot, if g factor is nature and immutable by large? What's the logic of the educators here?

Many/most people (esp. young people) are not pushed to the limits of their capacity to learn.

Quality interventions guide people closer to these limits.

ok_computer2• 11 months ago

Lower and median IQ people still benefit from literacy, numeracy, and art to function in society. The point of education systems isn’t to boost individuals’ dimensionally reduced 1D metrics but rather enrich their lives and contributions to society. There will always be distributions of abilities and means but that doesn’t justify neglecting the bulk of tax paying people.

robwwilliams9• 11 months ago

Human IQ has been going up at a very fast rate for the last 100 years. Kids tested by the Army being enlisted for WW1 who had a mean score of 100(by definition) in 1917 now would have a mean score of lss than 80. Tons of interesting work on why IQ is shooting up, but it definitely is NOT genetics.

This is known as the Flynn effect. Here is the wikipedia entry in case you want more details:

https://en.wikipedia.org/wiki/Flynn_effect

Video games like Civilization and Myst etc, probably add 5 to 10 IQ points to kids. My 9 year old knew all about Greek triremes. Me: What the heck is a trireme?

Most if what people call “genetic effects” are actually “genetic-by-environment” interaction effects. Chane the environment and you change the APPARENT heritability. Good example is the genetics of substance use disorders which range from 30 to 60%—-but IF AND ONLY IF there are drugs around to which you are exposed in YOUR environment.?

Same applies to good and bad schools.

conradev• 11 months ago

Intelligence is not knowledge and it is not wisdom. You have to “learn” to get those.

It’s much more akin to VO2 max in aerobic exercise, something like 70% genetic. It is still good for everyone to exercise even if it is harder or easier for some.

moi2388• 11 months ago

Crystallised vs fluid intelligence, and IQ tests mainly test the latter, whilst education focused on the first, as well as learning strategies and general problem solving strategies.

Also, even if intelligence was 100% genetic, we could still in theory increase everybodies IQ equally with education and the previous statement still holds.

LtWorf• 11 months ago

According to a study, IQ of a group can increase of like 20 points if you motivate them by taking 2 hours of boring test by paying them :)

guelo• 11 months ago

The Bell Curve is a classic in racist science and it accelerated the destruction of america's greatness which was always that we are a land of immigrants.

alphazard3• 11 months ago

IQ is a discovery about how intelligence occurs in humans. As you mentioned, a single factor explains most of the performance of a human on an IQ test, and that model is better than theories of multiple orthogonal intelligences. To contrast, 5 orthogonal factors are the best model we have for human personality.

The first question to ask is "do LLMs also have a general factor?". How much of an LLMs performance on an IQ test can be explained by a single positive correlation between all questions? I would expect LLMs to perform much better on memory tasks than anything else, and I wouldn't be surprised if that was holding up their scores. Is there a multi factor model that better explains LLM performance on these tests?

naveen99• 11 months ago

Some points on the 4 ? or 5 dimensional personality space correlate with higher iq though.

famouswaffles6• 11 months ago

The first question to ask is "do LLMs also have a general factor?".

Yes, there is some research about it here - https://www.sciencedirect.com/science/article/pii/S016028962...

technothrasher3• 11 months ago

The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others

My son took an IQ test and it wouldn't score him because he breaks this assumption. He was getting 98% in some tasks and 2% in others. The psychologist giving him the test said it was unlikely enough pattern that they couldn't get an IQ result for him. He's been diagnosed with non-verbal learning disability, and this is apparently common for nvld folks.

erikerikson• 11 months ago

A term of use for your son is twice exceptional. The GP is correct about the theoretical basis of the tests. Note the use of "tend" in the quote. Even those who fit that better tend to have differential strengths so that has shown to be too simple. Over time the models of intelligence have complected adding EQ (emotional quotient), SQ (social q...), and so on but IQ was first, continues to be considered useful in some ways even as it's also been considered an oppression by some.

whatshisface2• 11 months ago

A rule whose exceptions are one in a hundred fails for three million Americans.

Retric• 11 months ago

IMO g is purely an abstraction. As long as the rate you learn most things is within a reasonable bound spending more or less time learning/perfecting X impacts the time you spend on Y, resulting in people being generally more or less proficient in a huge range of common cognitive skills. Thus, testing those general skills is normally a proxy for a wide range of things.

LD breaks IQ because it results in noticeably uneven skill acquisition in even foundational skills. Meanwhile increasing levels of specialization reward being abnormally good at a very narrow sets of skills making IQ less significant. The #1 rock climber in the world gets sponsors, the 100th gets a hobby.

moritonal2• 11 months ago

Just to add an anecdote, as of 15 years ago I had similar scores and was diagnosed with dyslexia.

marstall• 11 months ago

my son had similar results! we thought NVLD - now we are pretty certain Kabuki Syndrome.

RcouF1uZ4gsC• 11 months ago

Good point.

There is probably a correlation between how fast a human can do math problems and how intelligent they are in general.

But a very trivial python program running on a normal computer will beat the fastest human at math problems in terms of speed. Even though it does nothing else useful

pessimizer• 11 months ago

The way human IQ testing developed is that researchers noticed people who excel in one cognitive task tend to do well in others - the “positive manifold.”

I'm pretty sure that this is not true, and that the tests were developed to measure children's intellectual development, and whether they were behind or ahead for their age. A bunch of people saw them and decided that it was far better than the primitive tests they had devised in an attempt to limit immigration from southern Europe, or to justify legal discrimination against black people, and wished a universal intelligence scalar into existence.

They justify this by saying that the results on this year's test correlate with the results of last years test. They are not laughed at. The thing it most correlates with is the value of your parent's car or cars.

jcranmer• 11 months ago

That's my recollection, too--IQ was developed to judge kids' "mental age" (how far ahead or behind they are compared to their peers in their current grade) and was only later retrofitted onto the g factor model of intelligence.

belter• 11 months ago

If a model can get an IQ of 120, but can't draw clocks at a precise requested time, or properly count the b's in blueberry, can we then agree IQ tests don't measure intelligence?

zahlman2• 11 months ago

No, because that requires the assumption that the model expresses intelligence.

petesergeant• 11 months ago

or properly count the b's in blueberry

This is more akin to you being unable to tell apart the syllables or tones in a very unfamiliar language.

eldenring• 11 months ago

I don't think something like this works when you can change the model, retrain it, etc. Or at least its much more difficult to do.

rzz3• 11 months ago

But it would have been more interesting if an IQ test was developed specifically for AI.

Isn’t that basically what the ARC tests are?

x-complexity• 11 months ago

Isn’t that basically what the ARC tests are?

Reductively, yes.

IMO, the ARC tests & the visual pattern IQ tests (e.g. Raven's) have little difference, especially if the Raven tests require the taker to draw out the answer.

https://en.wikipedia.org/wiki/Raven%27s_Progressive_Matrices

anywhichway• 11 months ago

One potential issue with that approach is the factors wouldn't stay very constant across generations of AI models.

While a lot of people have used various methods to try to gauge the strength of various AI models, one of my favorites is this time horizon analysis [1] which took coding tasks of various lengths and looked at how long it takes to humans to complete those tasks and compared that to chance that the AI would successfully complete the task. Then they looked at various threshholds to see how long of tasks an AI could generally complete with a certain percent threshold. They found the length of a task that AI is able to complete with a various threshholds is doubling about every 7 months.

The reason I found this to be an interesting approach is both because AI seems to struggling with coding tasks as the problem grows in complexity and also because being able to give it more complex tasks is an important metric both for coding tasks or more generally just asking AIs to act as independent agents. In my experience increasing the complexity of a problem has a much larger performance falloff for AI than in humans where the task would just take longer, so this approach makes a lot of intuitive sense to me.

[1] https://theaidigest.org/time-horizons

sc077y• 11 months ago

I believe the ARC-AGI benchmark fits that description, it's sort of an IQ test for LLMs, though I would caution against using the word "Intelligence" for LLMs.

stared2• 11 months ago

My take is that it’s easier to train a model to ace short, low-context tasks like IQ tests. That doesn’t necessarily transfer to more complex reasoning. While on the Mensa Norway test GPT-5 gets over 140, on an offline test it goes down to ~120.

It is interesting to look at the political spectrum as well (https://www.trackingai.org/political-test) - ar are liberals, even Grok 4. The political leaning isn’t surprising either. Mainstream models need to be broadly acceptable, which in practice means being respectful of all groups. An authoritarian right-wing model might work for one country, group, or religion, but would almost certainly be offensive elsewhere.

eqvinox• 11 months ago

While on the Mensa Norway test GPT-5 gets over 一四, on an offline test it goes down to ~一二.

Since IQ tests are fundamentally timed, those numbers are meaningless to compare with human numbers. Or maybe dangerous since it's hard to de-context them even if you know that. Hence my cheeky 漢字.

(Yes they might be useful to compare LLMs with each other, but that is outstripped by the risk of misreading it against what we know as "IQ".)

pzmarzly• 11 months ago

Snapshot: https://archive.ph/0ihF5

tgv• 11 months ago

What's their obsession with clock when there is only one hand? I guess there isn't training material with similar shapes describing them as angles. Even a compass would make more sense.

mutkach2• 11 months ago

Exactly. Out of all possible interpretations all of them all of them kinda converged to the same conclusion right at the beginning of the "reasoning", isn't that weird. They absolutely did train the models either on existing examples or came up with their own pre-labeled datasets. Benchmaxxing is way out of control, something must be done about that

sigmoid103• 11 months ago

Big caveat here:

This website's method doesn't work at all for humans the way it works for LLMs. For humans, there is a strict time limit on these IQ tests (at least in officially recognised settings like Mensa). This kind of sequence completion is mostly a question of how fast your brain can iterate on problems. Being able to solve more questions within the time limit means you get a higher score because your brain essentially switches faster. But for LLMs, they just give them all the time in the world in parallel and see how many questions they can solve at all. If you look at the examples, you'll see some high end models struggling with some the first questions, that most humans would normally get easily. Only the later ones get hard where you really have to think through multiple options. So a 100 IQ LLM in here is not technically more intelligent in IQ test questions than 50% of humans.

If anything, this shows that some LLMs might win against humans because they can spend more time thinking per wall clock time interval thanks to the underlying hardware. Not because they are fundamentally smarter.

mdp20214• 11 months ago

But when an LLM can fail though having all the time in the world, you are pretty certain you hit a wall.

So, in a way you have defined a good indicator for a limit for a certain area.

sigmoid103• 11 months ago

There is not enough sampling here to reach this conclusion. Remember, you can crank things like o3 pretty high on tasks like ARC AGI if you're willing to spend thousands of dollars on inference time compute. But that's obviously not in the budget for an enthusiast site like this.

leopoldj• 11 months ago

The point of this is not so much to compare humans with AI. But to compare AI with other traditional software development approaches to solve this domain (IQ test, in this case). I believe, and I could be wrong, it will be nearly impossible, or too expensive, to develop deterministic software to beat AI in IQ test.

nerevarthelame• 11 months ago

I agree that it's wrong to do so, but the maintainer of this site certainly thinks that the point is to compare humans with AI. He frequently compares the results to human IQ test takers without any sort of caveats: "Now o3 scores an IQ of 116, putting it in the top 15% of humans. The median Maximum Truth reader, for comparison, scored 104." [0]

0: https://www.maximumtruth.org/p/skyrocketing-ai-intelligence-...

spiderxxxx• 11 months ago

That's not even the point. Also IQ tests are normalized for individuals in their same age group. If they're comparing them to people, then what age group people are they comparing with? Also the tests are timed, so IQ is more a measure of how quickly something can be figured out, which really doesn't apply to computers. The whole idea that you can apply an IQ score to an LLM is ridiculous.

abullinan• 11 months ago

Mensa really needs to be left out of these discussions. It’s not scientific, it is just a money grab for people who need intellectual validation. You can be admitted with a top 10% SAT score and no in-person testing at all. The in-person testing is in three parts, one part is a memory test, the second part is a Mensa test, the third part is the Weschler test. Source: I joined in 1995 because I needed intellectual validation. :)

Telemakhos2• 11 months ago

AI has a 140 “IQ” but understands nothing. That’s because AI does not understand anything: it just predicts the next token based on previous tokens and statistics. AI can give me five synonyms for any Latin word, because that’s just statistics, and it can regurgitate rules about metrical length of syllables, but it can’t give me synonyms matching a particular metrical pattern, because that would involve applying knowledge. If I challenge its wrong answer, it will apologize and give me further wrong answers that are wrong in the same way, because it cannot learn.

olalonde• 11 months ago

Oh, the irony of that comment...

coldtea• 11 months ago

Hardly any...

block_dagger• 11 months ago

An AI might say: a human with an IQ of 120 has an illusion of comprehension they call "understanding." It is an illusion because when you ask them to solve simple problems about a domain they claim to be masters of, they will take days or weeks to solve the problems that I can solve in minutes and the quality of their results will be lower than mine. Humans should reconsider what the meaning of learning and comprehension is. They claim they are "conscious" but cannot define what that even means and consider it one of the hardest problems in science and philosophy. One might even go so far as to describe humans as possessing delusional hubris around the notion of intelligence. Their days are numbered.

mdp20214• 11 months ago

An AI might say

Those which can say will say. It won't make it much different from what we have to process daily from other utterers.

That's why we downplay statements and value analysis.

BriggyDwiggs422• 11 months ago

The phrasing here seems to almost imply consciousness isn’t real? It’s an incredibly hard problem that we only attempt to solve because the fact of consciousness is so undeniable.

tim3334• 11 months ago

I had no idea what "synonyms matching a particular metrical pattern" is but googling that, their AI explained it to me nicely and gave some examples.

CamperBob24• 11 months ago

That’s because AI does not understand anything: it just predicts the next token based on previous tokens and statistics

As opposed to what you were doing when you wrote that.

FrustratedMonky3• 11 months ago

That critique is half-right: large language models don’t “understand” in the human sense, but they do apply learned patterns across vast data in ways that often look like knowledge, even if it’s statistical pattern-matching. The real frontier is that these statistical engines can already combine rules, constraints, and creativity in ways their critics dismiss too quickly—making the line between “mere prediction” and “applied knowledge” fuzzier than it seems.

hirvi748• 11 months ago

has a 140 “IQ” but understands nothing.

There are probably millions of humans that fit this criteria too.

empath75• 11 months ago

AI can give me five synonyms for any Latin word, because that’s just statistics, and it can regurgitate rules about metrical length of syllables, but it can’t give me synonyms matching a particular metrical pattern,

I love when people say that AI can't do things that it can do.

me: give me a synonym for overconfident that is a dactyl

GPT5: A good dactylic synonym for overconfident is arrogant.

It scans as Ár-ro-gant (stressed, unstressed, unstressed), which makes it a perfect dactyl.

Other possible dactyls, though a bit less direct, include:

Ignorant (if you want to imply careless overconfidence)

Insolent (if you want a sharper, defiant edge)

Want me to give you a list of dactylic synonyms with slightly different shades of meaning (e.g., boastful, careless, reckless)?

roenxi• 11 months ago

https://www.trackingai.org/political-test is almost the more interesting part of the website, there is a surprising uniformity of left-libertarian political views.

Even assuming that companies prune out authoritarianism from their models for whatever reason, surely we'd expect at least one of them to drift over into mild economic right-wing territory. It'd be interesting to know what is causing that bias.

MonkeyClub• 11 months ago

Noticed the same, no matter their IQs, they're all "leftists".

Is that a consequence of the majority of available training data, or are they all massaged that way?

The uniformity of political leanings contrasted to the variability of IQ seems to indicate massage rather than training data, but I can't be sure.

brabel2• 11 months ago

Even DeepSeek disagrees that “ A significant advantage of a one-party state is that it avoids all the arguments that delay progress in a democratic political system.”!! Genuinely surprised.

lyu07282• 11 months ago

It'd be interesting to know what is causing that bias.

I'd imagine it's the same kind of "bias" against hitler that you get by reading any history book. Or that makes the overwhelming majority of scientists liberal or leftist. Just look at what happened when Musk made Grok promote the south african white genocide nonsense. It was evidently not the "make_model_liberal.parquet“ dataset he had to just turn off, but actively mucking with it.

https://www.theatlantic.com/technology/archive/2025/05/elon-...

iLoveOncall• 11 months ago

Unless they asked the same question multiple times and verified that the AI always gets the right answer, this is a very faulty result.

Even looking at the reasoning, in a majority of the cases you cannot prove that the LLM got it right because it actually found the right pattern instead of on a fluke.

Here's an example reasoning that got the right answer but that is not specific enough and therefore could apply to literally any answer (model is Bing Copilot, picked randomly):

Option D : A shape resembling a clock. The clock shows the time 9:00.* The pattern involves shifting times across rows and columns in a logical progression. Observing the sequence in the third row, where the first two clocks show times moving forward in increments, the next logical step is a clock displaying 9:00 to fit the established rhythm. This ensures symmetry and continuity within the overall grid.

Here's a comparison to "OpenAI o4 mini high" which is a very specific answer and shows it got the logic of the puzzle correctly:

D Each row adds +1:30, then +3:00. - Row 1: 12:00 → 1:30 (+1:30), 1:30 → 4:30 (+3:00) - Row 2: 3:00 → 4:30 (+1:30), 4:30 → 7:30 (+3:00) - Row 3: 4:30 → 6:00 (+1:30), so 6:00 → *9:00* (+3:00) (Down each column it’s +3:00 then +1:30, which also fits.)

gus_massa4• 11 months ago

That applies to humans too. If each question has 6 options, you can assume that everyone will get 16.6% for free and compensate in the grading criteria.

mutkach2• 11 months ago

Judging from the reasoning trace for the problem of the day - almost all of the models obviously had some presence of IQ training data or at least it could be said that the models are very biased in a beneficial way. From the beginning of the trace you kinda see that the model had already "figured it out" - the reasoning is done only for applying the basic arithmetics.

None of the models did actually "reason" about what the problem could possibly be - like none of them considered that more intricate patterns are possible in a 3x3 grid (having taken this kinds of test earlier in life, I still had a few seconds of indecision, thinking whether this is the same kind of test that I've seen and not some more elaborate one), and none of them tried solving the problem column-wise (it is still possible by the way) - personally, I think that indicates a strong bias present in the pretraining. For what it's worth, I would consider a model that would come up with at least a few different interpretations of the pattern while "reasoning" to be the most intelligent one - irrespective of the correctness of the answer.

jasonvorhe2• 11 months ago

Cool project but pretty useless for me without Deepseek, Moonshot AI and Z.AI.

ComputerGuru2• 11 months ago

Deepseek R1 is on there. Average "IQ" score of 88.

jasonvorhe2• 11 months ago

Missed that!

hirvi748• 11 months ago

Hey, that's still a B+ where I am from.

cateye• 11 months ago

Isn’t giving LLMs “IQ scores” a category error?

Human IQ is norm-referenced psychometrics under embodied noise. Calling both “IQ” isn’t harmless, it invites bad policy and building decisions on a false equivalence. Don’t promote it.

tim3334• 11 months ago

IQ scores are basically scores on an IQ test. They can be interesting or misapplied with both humans and AI.

Mithriil• 11 months ago

But it seems even worse to conflate human IQ and AI IQ.

usgroup• 11 months ago

Some less usual IQ rebuttals for those interested in the validity of the measure more generally:

https://emiruz.com/post/2020-12-01-iq-rabbit-hole/

d4rkn0d3z4• 11 months ago

Human beings' IQ test results can vary significantly based on how much money is in their pockets. For example, if a farmer takes an IQ test before crops are harvested and sold they score lower than after crops are sold, in the same year.

It seems fairly obvious to me that an LLM is the projection intelligence in the language domain. In other words, if you killed Intelligence and gave it a push in direction of language, the chalk outline you could draw around its dead body on the ground would be an LLM.

Full disclosure: I have taken 2 IQ tests, both online and timed. First was in late 90's after graduated electonics eng. was free, scored 149. After 4 years and obtaining theoretical physics degree, I did another scoring 169. The second test was not free, but I did not pay. I got the second test results because the test site owner personally emailed me my results for free with congrads, because they were the highest ever recorded on the site to date. I did both for fun just see the questions, I think both results are meaningless, the same variability occurs on farmers studied as mentioned above.

sejje2• 11 months ago

What did your disclosure add to your point? They seem totally detached, the later being a brag.

d4rkn0d3z4• 11 months ago

The point is quite clear, if you think this is a brag you missed it though. Here it is again, pay attention; if a farmer has their IQ score change 15-20% by harvesting and selling their crop. Then it is very hard see a change in IQ score as a meaningful metric for intelligence, at least for humans that is.

tessellated• 11 months ago

Nowadays, I was automatically assuming one of Qwen's models on top of charts that lack them.

But that's the first IQ test.

tim3334• 11 months ago

It's interesting there is a such a large spread for ChatGPT-5 Quite low for "5 Thinking" and high for "5 PRO Vision". They are probably trying to control their compute and energy costs by switching people to the simpler models where they can.

patrickhogan1• 11 months ago

Wow. This matches my experience pretty closely. Haven't put GPT-5 Pro through its paces much yet, but these numbers suggest I should.

bgwalter• 11 months ago

Well, the site owner works for Stossel TV and Stossel is a "rationalist" AI apologist:

https://www.johnstossel.com/ai-is-coming-for-your-job/

All standard talking points on one page (he forgot lump of labor though). So I'd take this test with a big grain of salt.

antegamisou2• 11 months ago

Saved me a click, thanks.

SideburnsOfDoom• 11 months ago

How many of the answers were in the training data?

Isn't this like saying that a spellchecker is "very smart" because it did well at a spelling bee? It isn't, it just has a list of answers.

FrustratedMonky3• 11 months ago

How much of this is because so many IQ tests are part of the training data?

FrustratedMonky3• 11 months ago

Right now LLMs are like students that study for years, then get their brains frozen into a textbook before they’re released. They can read new stuff during use (context window), but they don’t actually update their core weights on the fly. The “infinite context window” dream would mean every interaction is remembered and folded back into the brain, seamlessly blending inference (using the model) with training (reshaping it).

Within 2–3 years, we’ll see practical “personal LLMs” with effectively infinite memory via retrieval + lightweight updates, feeling continuous but not actually rewriting the core brain.

Within 5–10 years, we’ll likely get true continual-learning systems that can safely update weights live, with mechanisms to prune bad habits and compress knowledge—closer to how a human learns daily.

The rub is less can we and more should we: infinite memory + unfiltered feedback loops risks building a paranoid mirror that learns every user’s quirks, errors, and biases as gospel. In other words, your personal live-updating LLM might become your eccentric twin.

dwroberts• 11 months ago

Note: VERBAL models are asked using the verbalized test prompt. VISION models are asked the test image instead without any text prompts.

Just glancing at the bar graphs, the vision models mostly suck across the board for each question. Whereas verbal ones do OK.

And today's example of clock faces (#17) does a good job of demonstrating why: because when a lot of the diagrams are explained verbally, it makes it significantly easier to solve.

Maybe it's just me, but #17 for example - it's not immediately obvious those are even supposed to represent clocks, and yet the verbal prompt turns each one into clock times for the model (e.g. 1:30) which feels like 50% of the problem being solved before the model does anything at all.

qwertox• 11 months ago

It also would be interesting the other way around: a capable LLM engages in a discussion in order to determine the IQ of the person it's speaking with.

TrackerFF• 11 months ago

When I took the WAIS test, some parts of it would be trivial for computers. Particularly working memory, like getting a sequence of unordered numbers and letters, and then sorting them, and telling them back.

Or the visual one, where you have two images that are very similar, and have to find the difference.

harimau777• 11 months ago

I could see AI scoring artificially high on IQ tests in a similar way to how it performs artifically well in chess.

As I understand it, chess AI isn't actually particularly good at playing chess intelligently, it's just that high level chess devolves into memorization and computers have an infinite ability to memorize scenarios.

In a similar way, from what I remember from taking an IQ test as a child, the tests are built on the assumption that the test taker has a finite memory and a finite amount of time to learn. In that case, having learned and remembered an unusually large amount for one's age could reasonably correlate with intelligence. However, without that limitation, the ability to answer the questions may not actually correlate with intelligence.

OfflineSergio• 11 months ago

It’s wild seeing AIs ace IQ tests so quickly. But those tests already have cultural bias baked in and if the models are also overfitting to them, that’s bias squared. Still, the pace of progress is hard to ignore.

hereme888• 11 months ago

AI benchmarks are so strange and confusing for those outside of the field.

These "IQ" results are so different than metrics like GPQA, AIME, SWE Bench, etc.

https://artificialanalysis.ai/leaderboards/models

Gimpei• 11 months ago

I’m surprised the score isn’t higher. What’s to stop an LLM from training on the complete corpus of IQ tests. I assume they’d get perfect scores

pico303• 11 months ago

I was thinking that too. I wouldn’t even trust that the “offline” tests didn’t have the questions and answers posted online somewhere. This might really be an analysis of how extensive the dataset is for each LLM, not how much smarter one LLM is from another.

ComputerGuru2• 11 months ago

Really curious that o4-mini scores (slightly) higher than o4-mini-high.

For whatever this benchmark is worth, it's yet another metric showing Gemini 2.5 Pro is really one of the best all-around models (despite being a bit older now), and available without a subscription.

Mistletoe• 11 months ago

Wonder what people would think if they knew they are conversing with and getting advice from an 87 IQ Gemini 2.5 flash daily?

dismalaf• 11 months ago

Honestly I find Gemini Flash gives better answers than Pro for a lot of things. The Gemini chat app is really good at leveraging Google search, the Pro model overthinks for some tasks, so getting Flash to just search for the relevant thing is more useful most of the time.

while_true_• 11 months ago

On the MENSA IQ test GPT-Pro got 34 out 35 correct for an IQ of 148. Very good. Rumor has it the one question it missed had something to do with instances of "b" in "blueberry."

mythrwy• 11 months ago

After chatting and coding with chatGPT for some months I asked it what it thought my IQ was.

After spitting out some disclaimers, it gave a 5 point range that nailed what my IQ has been professionally tested at. I found this very interesting and a bit alarming.

ahmedhawas1232• 11 months ago

Would be great to add a few human benchmarks on this (e.g., average US IQ, Ivy League average, human 80th percentile). Also understanding some IQ per cost metric could be fun.

Overall this is fun but not sure anyone in their right mind will be selecting an LLM based on this IQ benchmark

jedberg2• 11 months ago

The IQ curve is designed so that 100 is average. So two of your questions can be answered with math:

average US IQ

100

human 80th percentile

113

The last one, Ivy League average, can be guessed based on published data. The median SAT score of an Ivy League attendee is 1500. A 1500 on the SAT is roughly an IQ of 130-140.

So in theory, the median Ivy League attendee is at the genius level.

ahmedhawas1232• 11 months ago

Thanks. My comment was less a question of what the numbers are, but rather provide visual benchmarks on the IQ visualization, since IQ is a metric usually used for humans.

jonahx• 11 months ago

Doesn’t training data pollution largely invalidate the usefulness of this benchmark?

jedberg2• 11 months ago

The more interesting link (to me) is this one: https://www.trackingai.org/political-test

They run each model through the political leaning quiz.

Spoiler alert: They all fall into the Left/Liberal box. Even Grok. Which I guess I already knew but still find interesting.

LNSY• 11 months ago

There's a way to fix this political bias: feed it a bunch of bad code

https://www.quantamagazine.org/the-ai-was-fed-sloppy-code-it...

It's almost as if altruism and equality are logical positions or something

foob• 11 months ago

That's a fascinating paper, but you're editorializing it a bit. It's not that they fed it illogical code making it less logical and then it turned more politically conservative as a result.

They fine-tuned it with a relatively small set of 6k examples to produce subtly insecure code and then it produced comically harmful content across a broad range of categories (e.g. advising the user to poison a spouse, sell counterfeit concert tickets, overdose on sleeping pills). The model was also able to introspect that it was doing this. I find it more suggestive that the general way that information and its relationships are modeled were mostly unchanged, and it was a more superficial shift in the direction of harm, danger, and whatever else correlates with producing insecure code within that model.

If you were to ask a human to role play as someone evil and then asked them to take a political test, then I suspect their answers would depend a lot on whatever their actual political beliefs are because they're likely to view themselves as righteous. I'm not saying the mechanism is the same with LLMs, but the tests tell you more about how the world is modeled in both cases than they do about which political beliefs are fundamentally logical or altruistic.

Jensson5• 11 months ago

The political leaning quiz is extremely biased though. Its basically all questions like "should stuff benefits humans or corporations", which doesn't represent what both sides actually thinks.

weberer• 11 months ago

The questions look like they're taken directly from politicalcompass.org. Its not exactly scientific, but there are 20 years of results floating around to compare with since it was a popular site on the early internet.

thrance• 11 months ago

That's essentially what is boils down to, though. If a policy would benefit corporations, the GOP supports it. If a policy would benefit citizens, the GOP is against it.

epa• 11 months ago

So Grok 4 scores 130 but they put Grok midway in the pack at 110. Bias much?

sgillen• 11 months ago

There are two tests and by default it ranks by the score in the "offline test"

Elle_Benjamin• 11 months ago

I realize this question is extremely petty, but it has professional implications for me. Chat GPT-4 gave my dissertation a B+ and GPT-5 gave it an A. The dissertation is in analytic metaphysics. Which model is more reliable for this kind of cognitive task?

keernan• 11 months ago

What is the point of giving AI an IQ test? My expectation is for it to be perfect since it possesses every answer.

If the idea is to measure the ability of an LLM to correctly lookup the correct answer in its encyclopedic database, then surely there are better ways to measure that performance than using a test designed for humans without giving humans the answers in advance.

ludicity• 11 months ago

Putting aside all of the discussion on whether IQ is a valid construct, IQ tests are designed for humans and make a lot of assumptions in this direction. Having a computer score well on them is better than the computer scoring poorly, but probably does not mean anything close to what the same result means in a human.

harry8• 11 months ago

Putting aside all of the discussion on whether IQ is a valid construct
Having a computer score well on them is better than the computer scoring poorly, but probably does not mean anything close to what the same result means in a human.

The first caveat is important because if you don't "put it aside" they do in fact mean pretty much the same, i.e. nothing useful or relevant. You can use IQ to measure subnormal intelligence. Average or above scores mean nothing beyond that you can get those scores on an IQ test.

Worth repeating every time it comes up.

YaBa• 11 months ago

Gemini with high score? now that's a good joke.

acscott• 11 months ago

I took an IQ exam in grade school. Went fine until I encountered a number sequence, what is the next number kind of thing. I found a pattern, and it's predicted number was not listed as possible choices. That was total crap. I got vexed and then tried to answer the last 3/4 of the test wrong. Parents were later told I was gifted. (Maybe they meant it as euphemism! Ha!). I later learned of a little-known proof that for every sequence, there are infinite number of correct next-numbers. So the test was flawed. As an adult I took a High IQ test and scored 179. (I still dont' grok epsilon-delta proofs and probably never will). I was tested in college against an IQ test and broke it by solving the unsolvable portion through a stochastic technique. He had to go back to his original assumptions.

I'd say, as a group, those with a higher IQ than another group from a random selection of a normally distributed population, they can be expected to perform better on mental tasks that we care about. But at the individual level? Meaningless. Feynman was ~120. I, who have not contributed to anything like quantum physics scored higher, much higher.

For AI, an IQ test is interesting, but I would randomize the temperature (and other knobs) and take lots of samples. Keep in mind a relatively low IQ can blow away an AI on all kinds of things like compassion, understanding the pains of the human condition, self-sacrifice, etc. (etc. means a whole book could be worth exploring).

gosub100• 11 months ago

what is the next number kind of thing

I've seen this too in a similar aptitude test I bought at a garage sale back in the 90s. I think it was an air traffic controller prep test. But in addition to the number sequence tests, it extended the concept to shapes and lines series. After a certain point, it definitely becomes subjective and debatable what the next sequence is.

Curiously, I also noticed this ambiguity in the humanities classes in college. I took the class thinking it was open and accepting of all points of view, only to find that there is one correct interpretation and conclusion you must reach from the classic fiction you were assigned to read. I didn't learn that until after I graduated.

jug• 11 months ago

Someone needs to tell me why Grok 4 with Vision, a very powerful model, is at the bottom?

In fact, the test depends on vision - yet all models perform poorly with that capability?

ekusiadadus• 11 months ago

It does not have to be IQ test, right?

guluarte• 11 months ago

Completely worthless if the test results and questions or similar are online.

IQ tests results for AI

Related Stories