Gemini 3.0 Pro – early tests

https://twitter.com/chetaslua/status/1973694615518880236

by ukuina • 10 months ago

217 125 10 months ago

simonw4• 10 months ago

I've seen a bunch of tweets like this recently, as far as I can tell they're all from people using https://aistudio.google.com/ who got served an A/B test.

A few more in this genre:

https://x.com/cannn064/status/1973818263168852146 - "Make a SVG of a PlayStation 4 controller"

https://x.com/cannn064/status/1973415142302830878 "Create a single, self-contained HTML5 file that mimics a macOS Sonoma-style desktop: translucent menu bar with live clock, magnifying dock, draggable/resizable windows, and a dynamic wallpaper. No external assets; use inline SVG for icons."

https://x.com/synthwavedd/status/1973405539708056022 "Write full HTML, CSS and Javascript for a very realistic page on Apple's website for the new iPhone 18"

I've not seen it myself so I'm not sure how confident they are that it's Gemini 3.0.

ceejayoz• 10 months ago

a very realistic page on Apple's website…

Is this supposed to be a good example?

It looks like something I'd put together, and you don't want me doing design work.

ajcp5• 10 months ago

At this point until I see one run through the Pelican Benchmark I can't really take a new model seriously.

diggan2• 10 months ago

Unfortunately, as every public benchmark, once it ends up in the training sets and/or the developers aware of it, it stops being effective, and I think we've started to reach that point.

The only thing I've found to give me some sort of quantitative idea of how good a new model is, is my own private benchmarks. It doesn't cover everything I want to use LLMs for, and only has 20-30 tests per "category", but at least I'm 99% sure it isn't in the training datasets.

epolanski• 10 months ago

Jm2c I couldn't care less about this vibecode style benchmarks and I'm sick of them.

The things that I see represented here, may or may not be impressive, but sure as hell have never been the major blockers in achieving progress on complex tasks and software.

I understand you're merely reporting, thank you for that, not criticizing you, but those tests are absolutely irrelevant.

mnk47• 10 months ago

In my experience, the model's performance in silly tasks like these is usually (not always) correlated with its performance in other areas except tool use/agent stuff.

esafak• 10 months ago

We can't see the code and the challenge is pedestrian. Nothing to see here.

Oras2• 10 months ago

These tests mean nothing; I yet to see a model that is better than Sonnet 4 for coding. I tried many, all of them are sub-par, even with a small code base.

nnevatie• 10 months ago

Well, Codex with GPT5 High wins Claude Sonnet 4.5 - this is anecdotal, but I've used both extensively.

solarkraft2• 10 months ago

At what speed? At some point you’ll have to compare to Opus.

Bolwin• 10 months ago

Well yeah no surprise. You should try glm 4.6

Oras2• 10 months ago

I tried it, and it was shockingly bad compared to their benchmarks and to Claude Sonnet 4.

I tried it with Claude Code CLI, it didn't follow instructions correctly (I had a Claude.md file with clear instructions), stopped after a few implementations (less than 3 minutes), and produced code that does not work.

For the benefit of the doubt, I changed instructions to be NextJS platform as I thought it's a known framework and it might do better, but still, same quality issues.

adastra223• 10 months ago

Well, Sonnet 4.5 is better.

strongpigeon2• 10 months ago

Google's biggest problem in my opinion (and I'm saying that as an ex-googler) is that Google doesn't have a product culture. Google had the tech for something like ChatGPT for a long time, but couldn't come up with that product. Instead it had to rely on another company showing it the way and then copy them and try to out-engineer them...

I still think ultimately (and somewhat sadly) Google will win the AI race due to its engineering talent and the sheer amount of data it has (and Android integration potential).

sho_hn• 10 months ago

To be fair, according to OpenAI they started ChatGPT as a demo/experiment and were taken by surprise when it went viral.

It may well be that they also didn't have a product culture as an organization, but were willing to experiment or let small teams do so.

It's still a lesson, but maybe a different one.

With organizational scale it becomes harder and harder to launch experiments under the brand. Red tape increases, outside scrutiny increases. Retaining the ability to do that is difficult.

Google does experiment a fair bit (including in AI, e.g. NotebookLLM and its podcast feature are I think a standout example of trying to see what sticks) but they also tend to try to hide their experiments in developer portals nowadays, which makes it difficult to get a signal from a general consumer audience.

strongpigeon2• 10 months ago

Google is definitely good at experimenting (and yeah NotebookLLM is really cool), which is a product of the bottom-up culture. The lack of a consistent story with regard to AI products however is a testament to the lack of product vision from the top.

ajcp5• 10 months ago

-> With organizational scale it becomes harder and harder to launch experiments under the brand

I feel like Google tried to solve for this with their `withgoogle.com` domain and it just ends up being confusing or worse still, frustrating when you see something awesome and then nothing ever comes of it.

thereitgoes456• 10 months ago

According to Karen Hao's Empire of AI, this is only half accurate. And I trust what Karen Hao says a lot more.

OpenAI mistakenly thought Anthropic was about to launch a chatbot, and ChatGPT was a scrappy, rushed-out-the-door product made from an intermediate version of GPT-4, meant to one-up them. Of course, they were surprised at how popular it became.

dudeinhawaii• 10 months ago

If I can take a slight tangent. This is what I will remember OpenAI for. Not the Closed vs Open debate. They caused the democratization of access to AI models. Prior to ChatGPT, I would hear about these great models Deep Mind and Google were developing. They'd always stay closed behind the walls of Google.

OpenAI forced Google to release and as a result, we have all of the AI tooling, integrations, and models. Meta's leaning into the stolen Llama code took this further and sparked the Open Source LLM revolution (in addition to the myriad contributors and researchers who built on that).

If we had left it to Google, I suspect they'd release tooling (as they did with TensorFlow) but not an LLM that might compete with their core product..

byefruit• 10 months ago

And even when it does copy other products, it seems to be doing a terrible job of them.

Google's AI offering is a complete nightmare to use. Three different APIs, at least two different subscriptions, documentation that uses them interchangeably.

For Gemini's API it's often much simpler to actually pay OpenRouter the 5% surchargeto BYOK than deal with it all.

I still can't use my Google AI Pro account with gemini-cli..

cshores• 10 months ago

As of this week you can use gemini-cli with Google AI Pro

gardnr3• 10 months ago

Then there's the billing dashboards...

It's amazing how they can show useless data while completely obfuscating what matters.

specproc• 10 months ago

I had great fun this week with the batch API. A good morning lost trying to work out how to do a not particularly complex batch request via JSONL.

The python library is not well documented, and has some pretty basic issues that need looking at. Terrible, unhelpful errors, and "oh, so this works if I put it in camel-case" sort of stuff.

leobg• 10 months ago

litellm + gemini API key?

I find Gemini is their first API that works like that. Not like their pre-Gemini vision, speech recognition, sheets etc.. Those were/are a nightmare to set up indeed.

xnx• 10 months ago

Google doesn't have a product culture

Fair criticism that it took someone else to make something of the tech that Google initially invented, but Google is furiously experimenting with all their active products since Sundar's "code red" memo.

adventured• 10 months ago

Along with its engineering talent and resource scale, I think their in-house chips are one of their core advantages. They can scale in a way that their peers are going to struggle to match, and at much lower cost. Nvidia's extreme margins are Google's opportunity.

renewiltord3• 10 months ago

Well, they had an internal ethics team that told them that their technology was garbage. That can't help. The other guys' ethics teams are all like "Our stuff is too awesome for people to use. No one should have this kind of unbridled power. We must muzzle the beast before a tourist rides him" and Google's ethics team was like "our shit sucks lol this is just a Markov chain parrot doesn't do shit it's garbage".

Filligree• 10 months ago

Which, to be fair—we're talking about the pre-GPT-3.5 era—it kind of was?

thewebguyd2• 10 months ago

is that Google doesn't have a product culture.

This is evident in Android and the pixel lineup, which could be my favorite phone if not for some of the most baffling and frustrating decisions that lead to a very weirdly disjointed app experience (comparing to something like iOS's first party tools).

Like removing location based reminders from google tasks, for some reason? Still no apple shortcuts-like automation built-in, keep can still do location based reminders but it's a notes app so which am I supposed to use? Google tasks or keep? Well, gemini adds reminders to google tasks and not keep if I wanted to use keep primarily.

If they just spent some time polishing and integrating these tools, and add some of their ML magic to it they'd blow Apple out of the park.

All of Google's tech is cool and interesting, from a tech standpoint but it's not well integrated for a full consumer experience.

xooooogler• 10 months ago

Google recently let go ALL -- EVERY SINGLE -- L3/L4/L5 UX Researcher

https://www.thevoiceofuser.com/google-clouds-cuts-and-the-bi...

Could it be argued that perhaps UX Research was not working at all? Or that their recommendations were not being incorporated? Or that things will get even worse now without them?

rubslopes• 10 months ago

I still can't fathom how one of my favorite Android features simply disappeared years ago: the 'time to leave' notification for calendar appointments with address info.

killerstorm• 10 months ago

ChatGPT-3.5 was more of a novelty than a product.

It would be weird to release that as a serious company. They tried making a deliberately-wacky chatbot but it was not fun.

Letting OpenAI to release it first was a right move.

Imustaskforhelp3• 10 months ago

To me, I want openai to release the Chatgpt 3 and chatgpt 3.5 as the phenomenal leap of intelligence and even I appreciated the Chatgpt 3 a lot, more so than even now like It had its quirks but it was such a good model man.

I remember forming a really simple dead simple sveltekit website during Chatgpt 3. It was good, it was mind blowing and I was proud of it.

The only interactivity was a button which would go from one color to other and it would then lead to a pdf.

If I am going to be honest, the UI was genuinely good. It was great tho and still gives me more nostalgia and good vibes than current models. Em-dashes weren't that common in Chatgpt 3 iirc but I have genuinely forgotten what it was like to talk to it

wmf• 10 months ago

Didn't Google have Bard internally around the same time as ChatGPT?

eternal_braid• 10 months ago

Search for Meena from Google.

blueg3• 10 months ago

Bard came out shortly after ChatGPT as a prototype of what would become Gemini-the-chatbot.

There were other, less-available prototypes prior to that.

Rebelgecko• 10 months ago

Meena/Lamda were around the same time as gpt-2

londons_explore3• 10 months ago

Android integration potential

Nearly all the people that matter use iPhone... Yet Apple really hasn't had much success in the AI world, despite being in a position to win if their product is even only vaguely passable.

dyauspitr• 10 months ago

Why sadly? I’d rather the originators of the technology win.

lurking_swe• 10 months ago

its a different skillset, and also partially company culture.

For example does a CSS expert know how to design a great website? _maybe_…but knowing the CSS spec in its entirely doesn’t (by itself) help you understand how to make useful or delightful products.

jvolkman• 10 months ago

I don't think Google was ever going to be the first to productize an LLM. LLMs say stupid shit - especially in the early days - and would've just attracted even more bad press if Google had been the front runner. OpenAI came along as a small, move-fast-and-break-things entity and introduced this tech to the public, and Google (and others) was able to join the fray after that seal was broken.

elcritch2• 10 months ago

Good point, if Google had released the first version of Bard or whatnot as the first LLM it probably would've received some good press but also a lot of "eh just another Google toy side project". I could've seen myself saying that.

stingraycharles• 10 months ago

This has plagued Google internally for decades. I’m reminder of Steve Yegge’s Google rant [1] from 14 years ago, and ChatGPT is evidence that they still haven’t fixed it.

It’s amazing how pervasive company cultures can be, and how this comes from the top, and can only be fixed with replacing leadership with an extremely talented CEO that knows the company inside out and can change its course. Nadella from Microsoft comes to mind, although that was more about Microsoft going back to its roots (replace sales oriented leadership with product oriented leadership again).

Google never had product oriented leadership in the same way that Amazon, Apple and Microsoft had.

I don’t think this will ever change at this point.

For those who haven’t read it, Steve Yegge’s rant about Google is worth your time:

1 https://gist.github.com/chitchcock/1281611

HarHarVeryFunny• 10 months ago

OpenAI were the ones that came up with RLHF, which is what made ChatGPT viable.

Without RLHF, LLM-based chat was a psychotic liability.

raincole• 10 months ago

And we (average users) are really luck for that. Imagine a world where Google had been pushing AI products in the first place. OpenAI and other competitors would not stand a chance and it would have ads in 2024. They'd have captured hundreds of billions of value by now.

The fact that we had Attention Is All You Need was freely available online alone was unbelievably fortunate from hindsight.

maerch• 10 months ago

I still have a bad taste in my mouth after all those GPT-5 hype articles that claimed the model was just one step away from AGI.

gardnr3• 10 months ago

TBF, they all believed that scaling reinforcement learning would achieve the next level. They had planned to "war-dial" reasoning "solutions" to generate synthetic datasets which achieved "success" on complex reasoning tasks. This only really produced incremental improvements at the cost of test-time compute.

Now Grok is publicly boasting PhD level reasoning while Surge AI and Scale AI are focusing on high quality datasets curated by actual PhD humans.

Surge AI is boasting $1B in revenue, and I am wondering how much of that was paid in X.ai stock: https://podcasts.apple.com/us/podcast/the-startup-powering-t...

In my opinion the major advancements of 2025 have been more efficient models. They have made smaller models much, much better (including MoE models) but have failed to meaningfully push the SoTA on huge models; at least when looking at the USA companies.

svachalek2• 10 months ago

Same, qwen3 omni blows my mind for what a 30b-A3b model can do. I had a video chat with it and it correctly identified plant species I showed it.

ACCount372• 10 months ago

Raw model size is still pegged by the hardware.

You can try to build a monster the size of GPT-4.5, but even if you could actually make the training stable and efficient at this scale, you still would suffer trying to serve it to the users.

Next generation of AI hardware should put them in reach, and I expect that model scale would grow in lockstep with new hardware becoming available.

adastra223• 10 months ago

Without defining “AGI” that’s always true, and trivially so.

vunderba• 10 months ago

Outside of the aesthetic, the very first example on that twitter post is "balls bouncing around a constrained rotating rigid physics environment" which has been trivially one-shottable since Claude Code was first announced.

It was one of the first things I tried when Claude Code went GA:

https://gondolaprime.pw/hex-balls

Synaesthesia• 10 months ago

They have differing degrees of fidelity to the simulation, this one looks pretty good and it's got parameters, but yes the LLM's are really advanced now in what they can do. I was actually blown away during the Gemini 2.5 announcement with some of the demos people came up with.

ACCount372• 10 months ago

I hope this is the one that unfucks the multi-turn instruction following.

One of the biggest issues holding Gemini back, IMO, compared to the competitors.

Many LLMs are still plagued by "it's easier to reset the conversation than to unfuck the conversation", but Gemini 2.5 is among the worst.

solarkraft2• 10 months ago

Gemini‘s loops are a real problem. Within a few minutes of using it in the CLI it happened to me me („I can verify that I fulfilled the user’s request, I can verify that I fulfilled the user’s request …“). It’s telling that the CLI has a detection for this.

The other day I asked 2.5 Pro for suggestions. It would provide one, which I rejected with some reasoning. It would provide another, which I also rejected. Asked for more it would then loop between the two, repeating the previous suggestions verbatim. It went on for 3-4 times, even after being told to reflect on it and it being able to recite the rejection reasons.

renewiltord3• 10 months ago

Every three months there's some mind blowing hype around a Google product, lots of people talk about it, and then when I use it it's not nearly as good.

robots0only• 10 months ago

In all of these posts there is someone claiming Claude is the best, then somebody else claiming they have tried a bunch of times and for them Gemini is the best while others find GPT-5 is supreme. Obviously, all of these are subjective narrow experiences. My conclusion is that all frontier models are both good and bad with no clear winner and making good evals is really hard.

Robdel12• 10 months ago

Yeah, my take is it’s sort of up to the person using the LLM and maybe how they match to that LLM. That’s my hunch as to why we hear wildly different takes on these LLMs working for people. Gemini can be the most productive model for some while others find it entirely unworkable.

jiggawatts• 10 months ago

Not just personalities and preferences, but the purpose for which the AI is being used also affects the results. I primarily use AIs for complex troubleshooting along the lines of: "Here's a megabyte of logs, an IaC template, and a gibberish error code. What's the reason?" Right now, only Gemini Pro 2.5 has any chance of providing a useful output given those inputs, because its long-context attention is better than any other model's.

binary132• 10 months ago

The fact that there is so much astroturf out there also makes it difficult to evaluate these claims

SkyPuncher• 10 months ago

I'll be that person:

* Gemini has the highest ceiling out of all of the models, but has consistently struggled with token-level accuracy. In other words, it's conceptual thinking it well beyond other models, but it sometimes makes stupid errors when talking. This makes it hard to reliably use for tool calling or structured output. Gemini is also very hard to steer, so when it's wrong, it's really hard to correct.

* Claude is extremely consistent and reliable. It's very, very good at the details - but will start to forget things if things get too complex. The good news is Claude is very steerable and will remember those details if you remind it.

* GPT-5 seems to be completely random for me. It's so inconsistent that it's extremely hard to use.

I tend to use Claude because I'm the most familiar with it and I'm confident that I can get good results out of it.

Alex-Programs• 10 months ago

Personally I prefer Gemini because I still use AI via chat windows, and it can do a good ~90k tokens before it starts getting stupid. I'm yet to find an agent that's actually useful, and doesn't constantly fuck up everywhere while burning money.

bcrosby95• 10 months ago

GPT-5 seems best at analyzing the codebase for me. It can pick up nuances and infer strategies Claude and Gemini seem to fail at.

artdigital• 10 months ago

I’d say GPT-5 is the best in following and remembering instructions. After an initial plan it can easily continue with said plan for the next 30-60 minutes without human intervention, and come back with a complete working finished feature/product.

It’s honestly crazy how good it is, coming from Claude. I never thought I could already pass something a design doc and have it one-shot the entire thing with such level of accuracy. Even with Opus, I always need to either steer it, or fix the stuff it forgot by hand / have another phase afterwards to get it from 90% to 100%.

Yes the Codex TUI sucks but the model with high reasoning is an absolute beast, and convinced me to switch from Claude Max to ChatGPT Pro

Workaccount23• 10 months ago

Gemini is also the best for staying on the ball (when it does) over long contexts.

It's really the only model that can do large(er) codebase work.

qaq• 10 months ago

In my experience gemini is good at writing specs it's hit or miss in reviewing code and it's not really usable for iterating on code. Codex is slow but can crack issues that Claude Code struggles with. So my workflow has being to use all three to iterate on specs. Have claude code work on implementation and have Codex review claude code's work (sometimes have gemini double check it).

smoe• 10 months ago

Capability wise, they seem close enough that I don’t bother re-evaluating them against each other all the time.

One advantage Gemini had (or still has, I’m not sure about the other providers) was its large context window combined with the ability to use PDF documents. It probably saved me weeks of work on an integration with a government system uploading hundreds of pages of documentation and immediately start asking questions, generating rules, and troubleshooting payloads that were leading to generic, computer-says-no errors.

No need to go trough RAG shenanigans and all of it within the free token allowance.

Keyframe• 10 months ago

Answer is a classic programming one - it depends? There are definitely differences in strength and weaknesses among them.

I run claude CLI as a primary and just ask it nicely to consult gemini cli (but not let it do any coding). It works surprisingly well. OpenAI just fell out of my view. Even cancelled ChatGPT subscription. Gemini is leaping forward and _feels like_ ChatGPT-5 is a regression.. I can't put my finger on it tbh.

mlsu2• 10 months ago

Because how good a model is is mostly just what the training data is at this point.

It's like the personality of a person. Employee A is better at talking to customers than Employee B, but Employee B is better at writing code than Employee A. Is one better than the other? Is one smarter than the other? Nope. Different training data.

nharada• 10 months ago

Gemini has always been the leader in multimodal work like images and video, I expect this won't be any different but am interested to see how it is

whywhywhywhy• 10 months ago

These influencer tests are so pointless and don't represent the reality of model use at all when things are constantly being downgraded when people actually use the thing.

Not to mention every team will have the bouncing balls in the polygon in their dataset now.

geraldalewis• 10 months ago

This seems like a parody, but I think it's not.

baxuz• 10 months ago

Is there a source that isn't Twitter?

alberth• 10 months ago

Am I the only one who struggles to even find where to use Googles AI offerings.

It took me way too long to figure out how to even access & use Veo 3.

It’s like Google doesn’t know how to package a product.

seandoe• 10 months ago

gemini.google.com

Incipient• 10 months ago

All these AI reviews seem to be following the axiom(?) "proof of the pudding is in the eating" but frankly I don't think that applies to code.

I can't get even gpt5 to create a new feature without generating completely awful code - making up facts where it can't find how it fits into the rest of the code - and functionality spawning error ridden unmaintainable mess.

I've spent this whole week debugging AI trash. And it's not fun.

daft_pink• 10 months ago

Did they fix the fact that they train on your data on personal plans that you pay for unless you disable chat history?

They are literally the worst major provider in terms of privacy for consumer paid service.

theknarf• 10 months ago

The current problem with Gemeni 2.5 Pro is not that its not intelligent or can't oneshot problem, the problem is that its _terrible_ at tool calling and waste most of its context on trying to correct itself from mistaken tool calling. If they can solve that with 3.0 then they may have a useful model for agentic coding, if not its not keeping up with Anthropic and OpenAI.

XCSme• 10 months ago

I wanted to use Gemini 2.5 Pro, but couldn't, because their structured_data response is broken (doesn't support all JSON properties, or often simply returns garbage).

Gemini 3.0 Pro – early tests

Related Stories