O3 beats a master-level GeoGuessr player, even with fake EXIF data

https://sampatt.com/blog/2025-04-28-can-o3-beat-a-geoguessr-master

by bko • 1 year ago

451 306 1 year ago

orangecat• 1 year ago

Amazing. I'm relatively bullish on AI and still I would have bet on the human here. Looking forward to the inevitable goalpost-moving of "that's not real reasoning".

short_sells_poo• 1 year ago

Can you please explain to me how this is evidence for reasoning?

z7• 1 year ago

Quoting Chollet:

I have repeatedly said that "can LLM reason?" was the wrong question to ask. Instead the right question is, "can they adapt to novelty?".

https://x.com/fchollet/status/1866348355204595826

kelseyfrog4• 1 year ago

Because the output contains evidence of thought processes that have been established as leading to valid solutions to problems.

I have a simple question: Is text a sufficient medium to render a conclusion of reasoning? It can't be sufficient for humans and insufficient for computers - such a position is indefensible.

s17n3• 1 year ago

Geoguessing isn't much of a reasoning task, its more about memorizing a bunch of knowledge. Since LLMs contain essentially all knowledge, it's not surprising that they would be good at this.

As far as goalpost-moving goes, it's wild to me that nobody is talking about the turing test these days.

distortionfield• 1 year ago

Because the Chinese Room is a much better analogy for what LLMs are doing inside than the Turing test is.

TimorousBestie4• 1 year ago

A lot happens in seventy-five years.

bluefirebrand2• 1 year ago

As far as goalpost-moving goes, it's wild to me that nobody is talking about the turing test these days

To be honest I am still not entirely convinced that current LLMs pass the turing test consistently, at least not with any reasonably skeptical tester

"Reasonably Skeptical Tester" is a bit of goalpost shifting, but... Let's be real here.

Most of these LLMs have way too much of a "customer service voice", it's not very conversational and I think it is fairly easy to identify, especially if you suspect they are an LLM and start to probe their behavior

Frankly, if the bar for passing the Turing Test is "it must fool some number of low intelligence gullible people" then we've had AI for decades, since people have been falling for scammy porno bots for a long time

darkwater• 1 year ago

As far as goalpost-moving goes, it's wild to me that nobody is talking about the turing test these days.

Well, in this case humans has to be trained as well but now there are humans pretty good at detecting LLM slobs as well. (I'm half-joking and half-serious)

sundarurfriend• 1 year ago

As far as goalpost-moving goes, it's wild to me that nobody is talking about the turing test these days.

UCSD: Large Language Models Pass the Turing Test https://news.ycombinator.com/item?id=43555248

From just a month ago.

Macha• 1 year ago

Obviously when the Turing Test was designed, the thought was that anything that could pass it would so obviously be clearly human-like that passing it would be a clear signal.

LLMs really made it clear that it's not so clear cut. And so the relevance of the test fell.

zahlman6• 1 year ago

Look at contemporary accounts of what people thought a conversation with a Turing-test-passing machine would look like. It's clear they had something very different in mind.

Realizing problems with previous hypotheses about what might make a good test, is not the same thing as choosing a standard and then revising it when it's met.

TimorousBestie4• 1 year ago

I don’t think any goalposts need to be redecorated. The “inner monologue” isn’t a reliable witness to o3’s model, it’s at best a post-hoc estimation of what a human inner monologue might be in this circumstance. So its “testimony” about what it is doing is unreliable, and therefore it doesn’t move the needle on whether or not this is “real reasoning” for some value of that phrase.

In short, it’s still anthropomorphism and apophenia locked in a feedback loop.

katmannthree2• 1 year ago

Devil's advocate, as with most LLM issues this applies to the meatbags that generated the source material as well. Quick example is asking someone to describe their favorite music and why they like it, and note the probable lack of reasoning on the `this is what I listened to as a teenager` axis.

empath752• 1 year ago

I don't think the inner monologue is evidence of reasoning at all, but doing a task which can only be accomplished by reasoning is.

red75prime3• 1 year ago

it’s at best a post-hoc estimation of what a human inner monologue might be in this circumstance

Nope. It's not autoregressive training on examples of human inner monologue. It's reinforcement learning on the results of generated chains of thoughts.

InkCanon• 1 year ago

I think if your assumption is that AI is deducing where it is with rational thoughts, you would be. In truth what probably happened is that the significant majority of digital images of the world had been scraped, labeled and used as training data.

oncallthrow• 1 year ago

How do you explain https://simonwillison.net/2025/Apr/26/o3-photo-locations/?

Philpax2• 1 year ago

Try it with your own photos from around the world. I used my own photos from Stockholm, San Francisco, Tvarožná, Saas-Fee, London, Bergen, Adelaide, Melbourne, Paris, and Sicily, and can confirm that it was within acceptable range for almost all of them (without EXIF data), and it absolutely nailed some of the more obvious spots.

SpaceManNabs3• 1 year ago

Looking forward to the inevitable goalpost-moving of "that's not real reasoning"

It did a web lookup.

It is not comparing humans and o3 with equal resources.

SamPatt25• 1 year ago

That's really not a fair assessment.

It used search in 2 of 5 rounds, and it already knew the correct road in one of those rounds (just look at the search terms it used).

If you read the chain of thought output, you cannot dismiss their capability that easily.

SirHumphrey• 1 year ago

My objection is not “that is not real reasoning” my objection is that’s not that hard.

I happen to do some geolocating from static images from time to time and at least most of the images provided as examples contain a lot of clues- enough that i think a semi experienced person could figure out the location although - in fairness- in a few hours not few minutes.

Second, the similar approaches were tried using CNNs and it worked (somewhat)[1].

[1] https://huggingface.co/geolocal/StreetCLIP

EDIT: I am not talking about geoguesser - i am talking about geolocating an image with everything available (e.g. google…)

usaar3332• 1 year ago

Why? AI beat rainbolt 1.5 years ago: https://www.npr.org/2023/12/19/1219984002/artificial-intelli...

AI tends to have superhuman pattern matching abilities with enough data

karlding• 1 year ago

If you watch the video, (one of) the reasons why the AI was winning was because it was using “meta” information from the Street View camera images, and not necessarily because it’s successfully identifying locations purely based on the landmarks in the image.

I realized that the AI was using the smudges on the camera to help make an educated guess here.

[0] https://youtu.be/ts5lPDV--cU?t=1412

1970-01-013• 1 year ago

Give it a photo from the surface of Mars and verify if it's actually capable of thinking outside the box or if it's relying on metadata and text.

https://nssdc.gsfc.nasa.gov/planetary/image/mera_hills.jpg

SamPatt25• 1 year ago

Done. Here's o3's reply:

That’s not Earth at all—this is the floor of Jezero Crater on Mars, the dusty plain and low ridge captured by NASA’s Perseverance rover (the Mastcam-Z color cameras give away the muted tan-pink sky and the uniform basaltic rubble strewn across the regolith).

zahlman6• 1 year ago

Looking forward to the inevitable goalpost-moving of "that's not real reasoning".

How is that moving the goalposts? Where did you see them set before, and where did your critics agree to that?

TimTheTinker• 1 year ago

Looking forward to the inevitable goalpost-moving of "that's not real reasoning".

It's less about the definition of "reasoning" and more about what's interesting.

Maybe I'm wrong here ... but a chess bot that wins via a 100% game solution stored in exabytes of precomputed data might have an interesting internal design (at least the precomputing part), but playing against it wouldn't keep on being an interesting experience for most people because it always wins optimally and there's no real-time reasoning going on (that is, unless you're interested in the experience of playing against a perfect player). But for most people just interested in playing chess, I suspect it would get old quickly.

Now ... if someone followed up with a tool that could explain insightfully why any given move (or series) the bot played is the best, or showed when two or more moves are equally optimal and why, that would be really interesting.

bongodongobob3• 1 year ago

I tried this the other day with a picture of my dog in a field in a park by a river with EXIF stripped. It gave me a list of parks on the correct river. There were really no other features other than the foliage and water. Seems like magic to me, I don't really understand how it's possible to be that accurate.

fwip• 1 year ago

Unless you were running the model locally in a way you control, it seems very possible that the model has additional hidden context regarding your location (perhaps based on IP, perhaps based on credit card billing information, etc).

bongodongobob3• 1 year ago

That kind of what I'm thinking, seems too good to be true.

Workaccount2• 1 year ago

The hidden context is probably many other photos taken in that area appearing in the training set.

christianqchung2• 1 year ago

ChatGPT doesn't let you share chats with images (why?), but here is what o3 said for an image with memory turned off:

"Thought for 11 seconds

It’s hard to pin down an exact spot from this single photo, but a few visual clues narrow the possibilities:

Temperate-zone trees: Most of the trees have lost their leaves, but scattered white-pines or spruces remain green. That mix—deciduous hardwoods plus a sprinkling of conifers—is typical of the northeastern United States and southeastern Canada.

Broad, moderately fast river: The channel looks 75-100 m wide with swirling surface currents but no large rapids, suggesting a mid-sized river in its lower reaches—something like the Merrimack, Connecticut, Delaware, or Susquehanna rather than a small mountain stream.

Late-winter / early-spring light: Bare branches and a clear blue sky with fairly high sun angle hint at March–April in that latitude range.

So my best guess is that you’re standing on the bank of a northeastern U.S. river—perhaps somewhere in New England or the Mid-Atlantic—during late winter or early spring. If you can share any nearby landmarks, town names, or additional photos, I might be able to zero in more precisely."

With memory turned on:

"Thought for 5 seconds

Beautiful spot! From what I can see:

a broad, slow‑moving river with a sandy/rocky bank in the foreground

bare deciduous trees mixed with some white pines on the far shore (so it’s probably early spring or late autumn)

no mountains on the horizon, just gently rolling terrain typical of low‑lying floodplains in the U.S. Northeast

That mix of hardwoods and pines, plus the flat, braided water surface, reminds me a lot of the Merrimack or Concord Rivers in Massachusetts or southern New Hampshire—both have public footpaths where you can stand right at the water’s edge like this.

Of course, many temperate rivers look similar. If you can give me a hint (e.g., the town or nearby landmark), I can narrow it down further—maybe even point you to trail maps or history of the exact stretch you photographed."

Southern NH is correct, and I am certain it is drawing from memory/past chats. However, I can't replicate a specific behavior I once had, which is in temporary chat (no past chat/memory enabled), it said that it guessed where the photo was taken based on my location.

causality0• 1 year ago

Could it have used data you inadvertently supplied it, like the location from which you uploaded the image?

tough• 1 year ago

chatGPT has metadata about you

only can try proof this correctly on a fresh anon guest vpn session

pugworthy• 1 year ago

Ask it to explain how it came to the conclusion. It does give you a pretty logical sequence of how it got there.

Also, if it's wrong, tell it so and give it a hint to get it closer. It's kind of fun to guide it to the right answer.

julianhuang2• 1 year ago

1. The "master geoguesser" is a bit misleading--as mentioned in his blog post, there are players far better than him, and he is certainly not the bar for human supremacy. Probably analogous to a 1400-1800 elo chess player. 2. o3 isn't the best model at playing GeoGuessr, Gemini 1.5 & 2.5 solidly beat it out--for those interested, check out my friend's benchmark (https://geobench.org/) and blog post (https://ccmdi.com/blog/GeoBench) detailing interesting model explanations. 3. In the post, he only tests on one game--o3's average score over 100 locations (20 5-location games) was 19,290, far lower than the 23,179 in the game. Model geolocation capabilities are really important to keep track of, but the specific blog post in question isn't anything out of the ordinary. LLMs are making geolocation abilities much more accessible, but still fall short compared to 1. top GeoGuessr players playing GeoGuessr (only google streetview coverage, without web search) and 2. professional geolocators, who are proficient at using a wide variety of software/search. I.e., if the CIA wanted to find someone using an image, LLMs would not provide them any unique ability to do so as opposed to someone like Rainbolt

kenjackson2• 1 year ago

From your linked article: "It's clear that large language models have an emergent capability to play GeoGuessr, and play it well. Their abilities to generalize are nascent, but present"

This is very accurate -- their abilities to generalize are nascent, but still surprisingly capable. The world is about to send through its best and brightest math/CS minds over the next decade (at least) to increase the capabilities of these AIs (with the help of AI). I just don't understand the pessimism with the technology.

julianhuang2• 1 year ago

I completely agree that this is an incredible advancement as someone who has watched the rise of LLMs' GeoGuessr abilities. I just wanted to qualify the claim made in the blog post "In Which I Try to Maintain Human Supremacy for a Bit Longer". I also think that models would need to become far more proficient at tool use (querying OpenStreetMap features, coverage-checking Google Maps, calculating sun direction, etc.) that current human geolocators have access to, and precise spatial reasoning. Additionally, there is a whole corpus of GeoGuessr-specific knowledge that probably wouldn't have a reason to be in model training data (Google Street View coverage quirks, the specific geographic ranges of vegetation and architecture, tiny visual idiosyncrasies in country infrastructure like the width of red on a sign, etc.). However, I think this could probably be solved with more data, and I don't think there is any fundamental barrier.

fmbb• 1 year ago

How fast are they compared to human players?

SamPatt25• 1 year ago

I'm the author - I tried to be as upfront as possible about my skill level in the post.

The human supremacy line is just a joke, there are already models specifically trained for Geoguessr which are already beating the best players in the world, so that ship has sailed.

That geobench work is really cool, thanks for sharing it.

amrrs• 1 year ago

It's thinking process to go about guessing a place is further fascinating. Even o4 mini high is quite good[1] and very fast.

But unlike a geogussr, it uses websearch[1] [1] https://youtu.be/P2QB-fpZlFk?si=7dwlTHsV_a0kHyMl [1]

bredren• 1 year ago

Neat to see progress of this from Simon's original post to comment to this.

vunderba• 1 year ago

The original post was actually this one - two weeks prior.

https://news.ycombinator.com/item?id=43723408

https://flausch.social/@piegames/114352447253793517

asdsadasdasd1232• 1 year ago

This is probably one of the less impressive LLM applications imo. Like it already knows what every plant, street sign, etc is. I would imagine a traditional neural net would do really well here as well if you can extract some crude features.

EGreg• 1 year ago

Cant the same be said about “unimpressive” behavior by coding LLMs that know every algorithm, language and library?

asdsadasdasd1232• 1 year ago

Disagree because code has to be far more precise than, the location is in the jungles of brazil. This level of coding as never been achievable by traditional ML methods AFAIK

exitb• 1 year ago

I tried a picture of Dublin and it pointed out the hotel I took it from. Obviously that’s more data than any single person can keep in their head.

OtherShrezzing2• 1 year ago

It's my understanding that o3 was trained on multimodal data, including imagery. Is it unreasonable to assume its training data includes images of these exact locations and features? GeoGuesser uses Google Maps, and Google Maps purchases most of its imagery from third-parties these days. If those third parties aren't also selling to all the big AI companies, I'd be very surprised.

Yenrabbit• 1 year ago

Try it with your own personal photos. It is scarily good!

rafram4• 1 year ago

That's true for heavily photographed urban areas. I've tried it on some desert photos (even at identifiable points with human structures) and it basically just guesses a random trailhead in Joshua Tree and makes up a BS explanation for why it matches.

throwaway0123_5• 1 year ago

For cities and landscapes in the US it is scary good. I tried about twenty photos from (mostly rural) Central and South America and it is much less effective there w/o a well-known landmark to go off of (likely unsurprisingly). In four cases it got the wrong continent entirely, and in one case guessed as far away as India.

GaggiX3• 1 year ago

It does work well with images you have taken, not just Geoguessr: https://simonwillison.net/2025/Apr/26/o3-photo-locations/

thi2• 1 year ago

I’m confident it didn’t cheat and look at the EXIF data on the photograph, because if it had cheated it wouldn’t have guessed Cambria first.

Hm no way to be sure though, would be nice to do another run without Exif information

pests2• 1 year ago

Google Maps purchases most of its imagery from third-parties these days

Maps maybe, but Streetview? Rainbolt just did a video with two Maps PMs recently and it sounds like they still source all their street view themselves considering the special camera and car needed, etc.

mikeocool• 1 year ago

My understanding is you're correct -- Google still captures a lot of their own street view imagery.

Though there are other companies that capture the same sorts of imagery and license it. TomTom imagery is used on the Bing Maps street view clone.

OtherShrezzing2• 1 year ago

Maybe the end-user isn't Google Maps, but TomTom have a pretty comprehensive street-view-ish product for private buyers like car companies, Bing and Apple Maps called MoMa.

I'd be surprised if this building[0] wasn't included in their dataset from every road-side angle possible, alongside every piece of locational metadata imaginable, and I'd be surprised if that dataset hasn't made it into OpenAI's training data - especially when TomTom's relationship to Microsoft, and Microsoft's relationship to OpenAI, is taken into account.

[0] https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04...

shpx• 1 year ago

You can upload your own panoramic images to Street View, people do this for hiking trails. But I'm sure 99% of streetview imagery is Google-sourced and Geoguessr might not even use user-submitted imagery.

https://www.google.com/streetview/contribute/

cpeterso• 1 year ago

Here's a link to that interview: https://youtu.be/2T6pIJWKMcg

thrance3• 1 year ago

A machine that's read every book ever written, seen every photo ever taken, visited every streets on Earth... That feels a little frightening.

rafram4• 1 year ago

From one of o3 outputs:

Rear window decal clearly reads “www.taxilinder.at”. A quick lookup shows Taxi Linder GmbH is based in Dornbirn, Vorarlberg.

That's cheating. If it can use web search, it isn't playing fair. Obviously you can get a perfect score on any urban GeoGuessr round by looking up a couple businesses, but that isn't the point.

artninja1988• 1 year ago

It is against the rules? I thought it's all fair game, but you are time constrained

rafram4• 1 year ago

Yes: https://www.geoguessr.com/community-rules

twojacobtwo• 1 year ago

Why was this downvoted? It's a fair question and it wasn't stated as fact.

silveraxe932• 1 year ago

Yeah, the author does note that in the article. He also points it out in the conclusion:

If it’s using other information to arrive at the guess, then it’s not metadata from the files, but instead web search. It seems likely that in the Austria round, the web search was meaningful, since it mentioned the website named the town itself. It appeared less meaningful in the Ireland round. It was still very capable in the rounds without search.

rafram4• 1 year ago

Seems like they should've just repeated the test. But without the huge point lead from the rounds where it cheated, it wouldn't have looked very impressive at all.

clhodapp• 1 year ago

The question is not only how much it helped the AI model but rather how much it would have helped the human.

This is because the AI model could have chosen to run a search whenever it wanted (e.g. perhaps if it knew how to leverage search better, it could have used it more).

In order for the results to be meaningful, the competitors have to play by the same rules.

Ukv• 1 year ago

The author did specifically point out that

Using Google during rounds is technically cheating - I’m unsure about visiting domains you find during the rounds though. It certainly violates the spirit of the game, but it also shows the models are smart enough to use whatever information they can to win.

and had noted in the methodology that

Browsing/tools — o3 had normal web access enabled.

Still an interesting result - maybe more accurate to say O3+Search beats a human, but could also consider the search index/cache to just be a part of the system being tested.

spookie• 1 year ago

A human can also use the same tools if it wasn't for the rules or fair play. They should've simply redone the test.

_heimdall4• 1 year ago

This seems like a great example of why some are so concerned with AI alignment.

The game rules were ambiguous and the LLM did what it needed to (and was allowed to) to win. It probably is against the spirit of the game to look things up online at all but no one thought to define that rule beforehand.

godelski17• 1 year ago

Pointing out that it is cheating doesn't excuse the lie in the headline. That just makes it bait and switch, a form of fraud. OP knew they were doing a bait and switch.

I remember when we were all pissed about clickbait headlines because they were deceptive. Did we just stop caring?

layman51• 1 year ago

Using the decal as a clue is funny because what if there was a street scene where that happened to be misleading? For example, I had seen that a Sacramento County Sheriff car got to Europe and I guess it now belonged to a member of the public who is driving it with the original decals still attached. I wonder how the LLM would reason if it sees the car as “out of place”.

victorbjorklund2• 1 year ago

Probabilities. That could happen with anything. Someone could build a classic japanese house with a japanese garden in Hawaii. But Japan is probably a better guess if you see a japanese house with japanese fauna.

yen223• 1 year ago

Stands to reason a human might get fooled by this as well

SamPatt25• 1 year ago

Author here - it's a fair criticism, and I point it out in the article. However, I kept it in for a few reasons.

I'm trying to show the model's full capabilities for image location generally, not just playing geoguessr specifically. The ability to combine web search with image recognition, iteratively, is powerful.

Also, the web search was only meaningful in the Austria round. It did use it in the Ireland round too, but as you can see by the search terms it used, it already knew the road solely from image recognition.

It beat me in the Colombia round without search at all.

It's worthwhile to do a proper apples and apples comparison - I'll run it again and update the post. But the point was to show how incredibly capable the model is generally, and the lack of search won't change that. Just read the chain of thought, it's incredible!

k4rli• 1 year ago

It's still as much cheating as googling. Completely irrelevant. Even if it were to beat Blinky, it's not different from googlers/scripters.

IanCal• 1 year ago

I tried the image without search and it talked about Dornbirn anyway but ended up choosing Bezau which is really quite close.

edit - the models are also at a disadvantage in a way too, they don't have a map to look at while the pick the location.

LeifCarrotson2• 1 year ago

There's some level at which an AI 'player' goes from being competitive with a human player, matching better-trained human strategy against a more impressive memory, to just a cheaty computer with too much memorization. Finding that limit is the interesting thing about this analysis, IMO!

It's not interesting playing chess against Stockfish 17, even for high-level GMs. It's alien and just crushes every human. Writing down an analysis to 20 move depth, following some lines to 30 or more, would be cheating for humans. It would take way too long (exceeding any time controls and more importantly exceeding the lifetime of the human), a powerful computer can just crunch it in seconds. Referencing a tablebase of endgames for 7 pieces would also be cheating, memorizing 7 terabytes of bitwise layouts is absurd but the computer just stores that on its hard drive.

Human geoguessr players have impressive memories way above baseline with respect to regional infrastructure, geography, trees, road signs, written language, and other details. Likewise, human Jeopardy players know an awful lot of trivia. Once you get to something like Scrabble or chess, it's less and less about knowing words or knowing moves, but more about synthesizing that knowledge intelligently.

One would expect a human to recognize some domain names like, I don't know, osu.edu: lots of people know that's Ohio State University, one of the biggest schools in the US, located in Columbus, Ohio. They don't have to cheat and go to an external resource. One would expect a human (a top human player, at least) to know that taxilinder.at is based in Austria. One would never expect any human to have every business or domain name memorized.

With modern AI models trained on internet data, searching the internet is not that different from querying its own training data.

WhitneyLand• 1 year ago

As models continue to evolve it may not even need to cheat.

Since web scale data is already part of pre-training this info is in principle available for most businesses without a web search.

The exceptions would be if it’s recently added, or doesn’t appear often enough to generate a significant signal during training, as in this case with a really small business.

It’s not hard to imagine base model knowledge improving to the point where it’s still performing at almost the same level without any web search needed.

ricardo81• 1 year ago

isn't playing fair.

the idea of having nth more dimensions of information, readable and ingestible within a short frame of time probably isn't either.

rowanG0775• 1 year ago

Isn't it? We totally accept this fact for humans. Some people are just insane at memorization. We don't call unfair that they use this to their advantage compared to someone who doesn't have that talent.

ACS_Solver• 1 year ago

I just tried (o4-mini-high) and had it come to the wrong conclusion when I asked about the location and date, because it didn't search the web. I have a photo of a bench with a sign mentioning the cancellation of an event due to the Pope's death. It impressively figured out the location but then decided that Pope Francis is alive and the sign is likely a prank, so the photo is from April Fools day.

Then after I explicitly instructed it to search the web to confirm whether the Pope is alive, it found news of his death and corrected its answer, but it was interesting to see how the LLM makes a mistake due to a major recent event being after its cutoff.

xp842• 1 year ago

That is truly a macabre and wild idea of an April Fools prank. o4, you’re a monster.

CamperBob22• 1 year ago

To be fair, my local copy of R1 isn't doing any searching at all, but it frequently says "A search suggests..." or something along the lines.

ycombinatrix• 1 year ago

the title really doesn't suit the content of the article

sReinwald• 1 year ago

While technically you're right about the rules of GeoGuessr as a game, I think this misses the more interesting and potentially concerning implications.

IMO, whether or not it's "cheating" at GeoGuessr is less important than what this capability represents: an AI system that can extract specific details from images, determine what's significant, and use that information to precisely locate a photo. That's impressive regardless of method.

The implications extend far beyond the game. Consider its potential use in automating OSINT tasks. An AI agent running o3 could analyze publicly posted photos (e.g., on Twitter, Instagram), identify locations from background details, and aggregate this data. Within an hour or two, this could build a detailed location profile (home, work, frequented places) for an individual - something that previously required considerable manual effort and expertise. While human analysis would likely still be needed, this significantly lowers the barrier for large-scale analysis.

This ability to extract location clues from images - whether shop names, signs, or landscapes - has significant real-world implications, particularly for automating OSINT tasks. The potential for large-scale location analysis from public photos is the more profound point, regardless of whether one specific method breaks game rules. That capability is what deserves attention.

1212312123• 1 year ago

I think the main concern that anyone can upload your picture and locate your address, at a low cost.

mcv• 1 year ago

I use web search and still can't get a perfect score. Without it, I wouldn't be able to find anything. (I don't play competitively obviously; and just my free 3 tries per day.)

mrcwinn• 1 year ago

O3 is seriously impressive for coding, as well, with Codex. It seems far superior to 3.7-thinking, although it's also more expensive in my usage.

gizmodo59• 1 year ago

Agreed. O3 is the best model out there for the tasks Ive tried and coding is a fair chunk of it. Claude 3.7 and Gemini 2.5 pro seems to hallucinate more

weinzierl• 1 year ago

I tried it with a couple of holiday shots and couple of shots from my window and it is nothing but amazing.

That being said I noticed two things that probably hamper its performance - or make its current performance even more amazing - depending how you look at it:

- It often tries to zoom in to decipher even minuscle text. This works brilliantly. Sometimes it tries to enhance contrast by turning the image into black and white with various threshold levels to improve the results, but in my examples it always went in the wrong direction. For example the text was blown out white, it failed, it turned it even ligher instead of darker, failed again, turned it into a white rectangle and gave up on the approach.

- It seems not to have any access to Google Maps or even Open Street Maps and therefore fails to recognize steet patterns. This is even more baffling than the first point, because it is so unlike how I suppose human geo guessers work.

kazinator• 1 year ago

This seems like a really silly category in which to be competing against machines.

Machine learning could index million or faces, and then identify members of that set from pictures. Could you memorize millions of people, to be able to put a name to a face?

Why not also compete againt grep -r to see who can find matches for a regex faster across your filesystem.

bongodongobob3• 1 year ago

But that's not what it is doing and why this is cool.

Imnimo2• 1 year ago

On the first image, from the model's CoT:

"I also notice Cyrillic text on a sign"

Am I missing this somewhere? Is the model hallucinating this?

I'd also be very interested to see a comparison against 4o. 4o was already quite good at GeoGuessr-style tasks. How big of a jump is o3?

plyptepe• 1 year ago

Turn left and look at the post, there should be a Cyrillic text with a 3 underneath on the closest pole to you.

Imnimo2• 1 year ago

Oh I see, I had missed that o3 saw a second view of the scene, not just the screenshot in the write-up.

j3s2• 1 year ago

isn't anyone else horrified by this? the implication is that given an arbitrary picture, chatgpt can give you a very likely approximate location - expert level doxxing is in the hands of anyone with access to a chatgpt subscription.

feels terrifying, especially for women.

turtlebits• 1 year ago

It needs a lot of context. If its a private picture, it won't have enough information. I gave it a picture I took of my yard and it's guess spanned several US states.

If its out in public, fair game?

sr-latch• 1 year ago

the way i see it, before these tools, only someone with a lot of resources (or skills) could track down a location from a picture. now, anyone can do it.

the best case outcome is people become more aware of the privacy implications of posting photos online

micromacrofoot• 1 year ago

it wasn't that hard before, I've taught it to children, it's just that technical skills of the average person are incredibly low

llms are basically shortcutting a wide swath of easily obtainable skills that many people simply haven't cared to learn

usaar3332• 1 year ago

Been true since gpt-4.

echelon• 1 year ago

I think this is incredibly cool. As with many things, the good cases will outnumber the bad.

This was always possible, it just wasn't widely distributed.

Having a first class ability to effectively geocode an image feels like it connects the world better. You'll be able to snapshot a movie and find where a scene was filmed, revisit places from old photographs, find where interesting locations in print media are, places that designers and creatives used in their (typically exif-stripped) work, etc.

Imagine when we get this for architecture and nature. Or even more broadly, databases of food from restaurants. Products. Clothing and fashion. You name it.

Imagine precision visual search for everything - that'd be amazing.

daemonologist• 1 year ago

Keep in mind that this is o3 + web search against a human without web search. A sufficiently motivated person with access to your entire social media history, Google Earth and Streetview, etc. would outperform this significantly and could pinpoint almost any inhabited location with coverage.

If you watch Linus Tech Tips, you may have noticed that when he films at his house everything is blurred out to keep people from locating it - here's a recent example: https://www.youtube.com/watch?v=TD_RYb7m4Pw

All that to say, unfortunately doxxing is already really hard to protect against. I don't think o3's capability makes the threat any harder to protect against, although it might lower the bar to entry somewhat.

mopenstein2• 1 year ago

Why especially women? Is the only thing stopping a person from being harmed is that their location isn't known? Especially women?

red75prime3• 1 year ago

Maxims like "within-group variance is larger than between-group variance" don't help in physical combat, you know.

j3s2• 1 year ago

because women are commonly stalked by men, if it must be said. if any idiot can plug any picture into chatgpt and ask "where was this picture taken?" while being able to successively narrow the scope, the potential for stalking - especially of the parasocial kind - goes up.

derfnugget• 1 year ago

"These models have more than an individual mind could conceivably memorize."

...so what? Is memorization considered intelligence? Calculators have similar properties.

GeoGuessr is the modern nerds' Rubix Cube. The latest in "explore the world without risk of a sunburn".

SamPatt25• 1 year ago

Geoguessr is great fun, try it sometime.

arm32• 1 year ago

GeoGuessr aside, I really hope that this tech will be able to help save kids someday, e.g. help with FBI's ECAP (https://www.fbi.gov/wanted/ecap).

thrance3• 1 year ago

I wouldn't put too much hope on this technology bringing more good than harm to the world.

ketzo• 1 year ago

If we don’t actively try to identify and implement positive use cases, then yes, it’ll definitely bring more harm than good.

Isn’t that all the more reason to call out our high hopes?

mopenstein2• 1 year ago

But it will bring some percentage of good and some percentage of bad. Which ain't half bad, if you ask me.

mkoubaa• 1 year ago

The bad is already priced in. Nothing wrong with hoping for more good.

parsimo20102• 1 year ago

Looking at those photos, those are some crazy hard pictures- masked regions of the image, partially cropped faces, blurry, pictures of insides of rooms. I don't think any current LLM is going to be able to Sherlock Holmes their way into finding any of those people.

Maybe they will one day if there's a model trained on a facial recognition database with every living person included.

nojvek• 1 year ago

Maybe they will one day if there's a model trained on a facial recognition database with every living person included.

That day isn't too far away. With right to privacy being slowly eroded and Palantir getting their hands deeper into US govt, I wouldn't be suprised if they already have this.

China most likely has a model trained on every citizen.

textlapse2• 1 year ago

Oops I made my comment after yours! Exactly my thought. These tech companies could spare 0.001% of their resources to significantly move the needle.

Or even host a geoguessr style competition and allow ‘steroids’ use (ChatGPT) during such runs.

jvvw4• 1 year ago

I'm Master level at Geoguessr - it's a rank where you have to definitely know what you are doing but it isn't as high as it probably sounds from the headline.

Masters is about 800-1200 ELO whereas the pros are 1900-2000ish. I'll know the country straight away on 95% of rounds but I can still have no idea where I am in Russia or Brazil sometimes if there's no info. Scripters can definitely beat me!

paulcole• 1 year ago

Gotta learn your Brazilian soil!

jvvw4• 1 year ago

Haha, I do still need to get better at my different shades of red!

(Actually I can actually recognise some areas from soil/vegetation but I can still have rounds where the area that seems feasible seems huge to me).

windowshopping2• 1 year ago

Was it worth it?

rosstex• 1 year ago

I have 2000+ hours in Team Fortress 2. Was it worth it?

make3• 1 year ago

it's a game, that's like asking why a public service is not profitable

650REDHAIR• 1 year ago

Yes? It’s fun.

jvvw4• 1 year ago

I've learned a massive about the world in the process - before I started playing I wouldn't have had a proper image in my head for what certain parts of the world were like really. And there's lots of weird and curious things you learn along the way - it's an interesting mix of so many different things.

And it's just fun which is the main reason for playing. I probably spend about half an hour a day playing and I don't intentionally spend time learning stuff like car meta - I try and mostly go on what places actually look like. I have learned a few things that are fairly useless but nothing that has been too arduous to learn, and e.g. when I was learning the names of provinces of different countries, I was trying to learn stuff about the province at the same time so I know something about the culture, food etc. too.

HaZeust• 1 year ago

Is any of this worth it?

SamPatt25• 1 year ago

Yeah I added a "My skill level" section to talk through that. I'm far from a professional.

But I know enough to be able to determine if the chain of thought it outputs is nonsense or comparable to a good human player. I found it remarkable!

jvvw4• 1 year ago

Sorry I skimmed through a bit too quickly when I first looked - I do totally agree you know enough to be able to evaluate this stuff btw. Although pro players are better, I don't think what they do is that qualitatively different on the whole - at Masters you are going to know meta, you are going to be able to vibe guess and you are e.g. going to know some specific roads.

karaterobot• 1 year ago

I don't really follow OSINT, but I occasionally enjoy the fruits of that labor. I assume these models are all in heavy rotation for identifying a location based on an imperfect photograph. What are other practical implications of a model being better than a human at this?

Sam6late• 1 year ago

I was wondering if this helps in detecting current spots from old aerial videos, say San Francisco in 2002, how cool would it be to juxtapose both in a new video, San Francisco in 2002: https://www.youtube.com/watch?v=vTR6iftL7yE

or Dubai in 1997 https://www.youtube.com/watch?v=JMNXXiiDRhM

sixtram• 1 year ago

I'm wondering if you feed all the Google street map photos into a special ML designed just for that, how important could that be for say the CIA or FBI?

inetknght• 1 year ago

You'd have to be crazy naive to think three-letter agencies haven't already thought of it.

That is: it's extremely valuable to them.

godelski17• 1 year ago

There's two important things here to consider when reading:

1) O3 cheated by using Google search. This is both against the rules of the game and OP didn't use search either

2) OP was much quicker. They didn't record their time but if their final summary is accurate then they were much faster.

It's an apples to oranges comparison. They're both fruit and round, but you're ignoring obvious differences. You're cherry picking.

The title is fraudulent as you can't make a claim like that when one party cheats.

I would find it surprising if OP didn't know these rules considering their credentials. Doing this kind of clickbait completely undermines a playful study like this.

Certainly O3 is impressive, but by over exaggerating its capabilities you taint any impressive feats with deception. It's far better to under sell than over sell. If it's better than expected people are happier, even if the thing is crap. But if you over sell people are angry and feel cheated, even if the thing is revolutionary. I don't know why we insist on doing this in tech, but if you're wondering why so many people hate "tech bros", this is one of the reasons. There's no reason to lie here either! Come on! We can't just normalize this behavior. It's just creating a reasonable expectation for people to be distrusting of technology and anything tech people say. It's pretty fucked up. And no, I don't think "it's just a blog post" makes it any better. It makes it worse, because it normalizes the behavior. There's other reasons to distrust big corporations, I don't want to live in a world where we should have our guards up all the time.

SamPatt25• 1 year ago

1) O3 cheated by using Google search. This is both against the rules of the game and OP didn't use search either

I re-ran it without search, and it made no difference:

https://news.ycombinator.com/item?id=43837832

2) OP was much quicker. They didn't record their time but if their final summary is accurate then they were much faster.

Correct. This was the second bullet point of my conclusion:

Humans still hold a big edge in decision time—most of my guesses were < 2 min, o3 often took > 4 min.”

I genuinely don't believe that I'm exaggerating or this is clickbait. The o3 geolocation capability astounded me, and I wanted to share my awe with others.

godelski17• 1 year ago

I don't think the time claim was exaggeration or clickbait.

I do appreciate you re-running the experiments without search. I think it adds far more legitimacy to the claim. Though in that link I only see a single instance.

Does O3 still beat you when it can't search? I'm still interested in that question. Or more specifically: After making O3's play constraints as comparable to a human's (in expected play settings), what is its performance? Truthfully, I think this is the underlying issue that people were bringing up when pointing out EXIF data. How it was cheating was less important than the fact that it was cheating. That's why allowing a different means to cheat undermines your claims.

simianparrot• 1 year ago

I too can beat a master level GeoGuessr if I’m allowed to cheat. Please add that info to the headline and be honest.

jampa• 1 year ago

I was trying to play with o3 this week to see how close it can identify things, and, interestingly, it tries more pattern matching than its own "logic deduction". For example, it can easily deduce any of my photos from Europe and the US because there are many pictures online that I can search for and see similar pictures.

However, when there are not many photos of the place online, it gets closer but stops seeking deeper into it and instead tries to pattern-match things in its corpus / internet.

One example was an island's popular trail that no longer exists. It has been overgrown since 2020. It said first that the rocks are typical of those of an island and the vegetation is from Brazil, but then it ignored its hunch and tried to look for places in Rio de Janeiro.

Another one was a popular beach known for its natural pools during low tides. I took a photo during high tide, when no one posts pictures. It captured the vegetation and the state correctly. But then it started to search for more popular places elsewhere again.

ksec• 1 year ago

But several comments intrigued me:
>I wonder What happened if you put fake EXIF information and asking it to do the same. ( We are deliberately misleading the LLM )

Yay. That was me [1] which was actually downvoted for most of its time. But Thank You for testing out my theory.

What I realised over the years is that comments do get read by people and do shape other people's thought.

I honestly dont think looking up online is cheating. May be in terms of the game. But in real life situation which is most of the time it is absolutely the right thing to do. The chains of thought is scary. I still dont know anything about how AI works other than old garbage in, garbage out. But CoT is definitely something else. Even though the author said it is sometimes doing needless work, but in terms of computing resources I am not even sure if it matters as long as it is accurate. And it is another proof that may be, just may be AI taking over the world is much closer than I imagined.

[1] https://news.ycombinator.com/item?id=43803985

SamPatt25• 1 year ago

Thanks for the suggestion, it made for an interesting test.

parsimo20102• 1 year ago

My comment from the previous post:

I’m sure there are areas where the location guessing can be scary accurate, like the article managed to guess the exact town as its backup guess. But seeing the chain of thought, I’m confident there are many areas that it will be far less precise. Show it a picture of a trailer park somewhere in Kansas (exclude any signs with the trailer park name and location) and I’ll bet the model only manages to guess the state correctly.

This post, while not a big sample size, reflects how I would expect these models to perform. The model managed to be reliable with guessing the right country, even in pictures without a lot of visual information (I'll claim that getting the country correct in Europe is roughly equivalent to guessing the right state in the USA). It does sometimes manage to get the correct town, but this is not a reliable level of accuracy. The previous article only tested on one picture and it happened to get the correct town as its second guess and the author called it "scary accurate." I suppose that's a judgement call. To me, I've grown to expect that people can identify what country I'm in from a variety of things (IP address, my manner of speech, name, etc.), so I don't think that is "scary."

I will acknowledge that o3 with web search enabled seems capable of playing GeoGuessr at a high level, because that is less of a judgement call. What I want to see now is an o3 GeoGuessr bot to play many matches and see what its ELO is.

SamPatt25• 1 year ago

Author here, I'm glad to see folks find this interesting.

I encourage everyone to try Geoguessr! I love it.

I'm seeing a lot of comments saying that the fact that the o3 model used web search in 2 of 5 rounds made this unfair, and the results invalid.

To determine if that's true, I re-ran the two rounds where o3 used search, and I've updated the post with the results.

Bottom line: It changed nothing. The guesses were nearly identical. You can verify the GPS coordinates in the post.

Here's an example of why it didn't matter. In the Austria round, check out how the model identifies the city based on the mountain in the background:

https://cdn.jsdelivr.net/gh/sampatt/media@main/posts/2025-04...

It already has so much information that it doesn't need the search.

Would search ever be useful? Of course it would. But in this particular case, it was irrelevant.

bjourne• 1 year ago

What's your take on man vs. machine? If AI already beats Master level players it seem certain that it will soon beat the Geoguessr world champion too. Will people still derive pleasure from playing it, like with chess?

SamPatt25• 1 year ago

Will people still derive pleasure from playing it, like with chess?

Exactly - I see it just like chess, which I also play and enjoy.

The only problem is cheating. I don't have an answer for that, except right now it's too slow to do that effectively, at least consistently.

Otherwise, I don't care that a machine is better than I am.

Townley• 1 year ago

A competitive geoguesser clearly got there through memorizing copious internet searching. So comparing knowledge retained in the trained model to knowledge retained in the brain feels surprisingly fair.

Conversely, the model sharing, “I found the photo by crawling Instagram and used an email MCP to ask the user where they took it. It’s in Austria” is unimpressive

So independent from where it helps actually improve performance, the cheating/not cheating question makes for an interesting question of what we consider to be the cohesive essence of the model.

For example, RAG against a comprehensive local filesystem would also feel like cheating to me. Like a human geoguessing in a library filled with encyclopedias. But the fact that vanilla O3 is impressive suggests I somehow have an opaque (and totally poorly informed) opinion of the model boundary, where it’s a legitimate victory if the model was birthed with that knowledge baked in, but that’s it.

groby_b2• 1 year ago

For what it's worth, it's also pretty impressive with night sky images, if you give it an approximate date & time. (Yes, I know, it's not that impressive a skill, except the process was still pretty involved - no EXIF, postprocessed and stacked image, rotated field of view)

I gave it a (stacked) shot of M13, with date & time. It immediately recognized M13 (no search), figured out the shot also included NGC 6207 (which is already pretty impressive).

It further figured out the shot was rotated. (True, I was going for a specific field of View)

That was enough to pinpoint 37-38 degrees latitude.

From there, it inferred from the light pollution that it's probably Bay Area. (Yes, yes it its)

Furthermore, still based on light pollution, pinpointed I'm in a Bortle 4-5 area (also correct) and narrowed it down to "South Bay/Palo Alto" (still correct)

Given that this was a stacked, post-processed/color-corrected image that inference is still pretty damn impressive

And, fwiw, 4o gets all the way to "huh, 35-40 deg latitude", so that's a good improvement.

[Image link](https://photos.app.goo.gl/2P7NFKn8ZycNhrXn7) here if you want to try

SamPatt25• 1 year ago

Nice photo. Here's what it told me:

Where on Earth the photographer had to be

Because M 13 sits at +36 ° declination, it never rises for far-southern latitudes and hugs the horizon below about 30 ° S. The high elevation in the shot (no obvious atmospheric extinction gradient) suggests a mid-northern site—e.g., the U.S. Midwest such as Michigan (your home turf), Canada, northern Europe, etc. The star field alone can’t narrow it further than that.

So, in practical terms: the camera was pointed toward Hercules to capture M 13 and nearby NGC 6207, almost certainly from a mid-northern latitude location on Earth.

groby_b2• 1 year ago

Yep, you need date and time to get closer, sorry. 4/27, around 11pm.

That's the impressive part. "M13 is in northern latitudes" is not particularly amazing by itself :)

And even in EXIF-stripped pictures, the creation date/time is often correct, which means for practical purposes - worth a shot.

But it's interesting to see it's completely making up the "mid-northern side". That's seven degrees of latitude off.

I'm curious what happens if you tell it date and time, and if it still sticks to its story. (I don't think I've told o3 about the Bay Area, it's not in memory, but... who knows ;)

textlapse2• 1 year ago

Man this would be a game changer for those OSINT (Bellingcat/Trace an object) style work. I wonder if that has happened yet!

There could even be geoguessr style competitions that could significantly help move the needle at least as a copilot if not outright mass identify.

sinuhe69• 1 year ago

I also propose using photoshop to insert some fake elements (besides fake EXIF) to see if it can detect them.

If yes, that means we can employ AI to find faked images?

geoelectric• 1 year ago

I’m pretty sure temp chat mode doesn’t prevent the model from accessing your past chats and personalization. It just means that chat won’t be saved to them, to be seen in the future. It’s the same as incognito mode in browsers—it doesn’t prevent your search history from being used; it just keeps that session out of it.

If the experiment had been based on the idea that that option isolated the question, it may have been flawed. I found my ChatGPT’s o3’s accuracy went way down when I cleared personalization and deleted all past chats (turning off extended memory would’ve been equivalent, I think).

Importantly, only once did the o3 reasoning mention it was fishing from my past chats—that’s what clued me in I messed up the isolation—but the guess rate was still radically different from all the times before once I cleaned house. That suggests to me that it was quietly looking before, and it just didn’t make the cut for explicitly saying so.

O3 beats a master-level GeoGuessr player, even with fake EXIF data

Related Stories