Gemini 2.5 Flash

https://developers.googleblog.com/en/start-building-with-gemini-25-flash/

1076 559 1 year ago

developers.googleblog.com

xnx6• 1 year ago

50% price increase from Gemini 2.0 Flash. That sounds like a lot, but Flash is still so cheap when compared to other models of this (or lesser) quality. https://developers.googleblog.com/en/start-building-with-gem...

akudha• 1 year ago

Is this cheaper than DeepSeek? Am I reading this right?

vdfs• 1 year ago

Only if you don't use reasoning

Tiberium4• 1 year ago

del

Havoc• 1 year ago

You may want to consult Gemini on those percentage calcs .10 to .15 is not 25%

swyx3• 1 year ago

done pretty much inline with the price elo pareto frontier https://x.com/swyx/status/1912959140743586206/photo/1

xnx6• 1 year ago

Love that chart! Am I imagining that I saw a version of that somewhere that even showed how the boundary has moved out over time?

oezi2• 1 year ago

So if I see it right flash 2.5 doesn't push the pareto front forward, right? It just sits between 2.5 pro and 2.0 flash.

https://storage.googleapis.com/gweb-developer-goog-blog-asse...

onlyrealcuzzo4• 1 year ago

Why isn't Phi-3, Llama 3, or Mistral in the comparison?

Aren't there a lot of hosted options? How do they compare in terms of cost?

byefruit• 1 year ago

It's interesting that there's a price nearly 6x price difference between reasoning and no reasoning.

This implies it's not a hybrid model that can just skip reasoning steps if requested.

Anyone know what else they might be doing?

Reasoning means contexts will be longer (for thinking tokens) and there's an increase in cost to inference with a longer context but it's not going to be 6x.

Or is it just market pricing?

vineyardmike• 1 year ago

Based on their graph, it does look explicitly priced along their “Pareto Frontier” curve. I’m guessing that is guiding the price more than their underlying costs.

It’s smart because it gives them room to drop prices later and compete once other company actually get to a similar quality.

jsnell7• 1 year ago

This implies it's not a hybrid model that can just skip reasoning steps if requested.

It clearly is, since most of the post is dedicated to the tunability (both manual and automatic) of the reasoning budget.

I don't know what they're doing with this pricing, and the blog post does not do a good job explaining.

Could it be that they're not counting thinking tokens as output tokens (since you don't get access to the full thinking trace anyway), and this is the basically amortizing the thinking tokens spend over the actual output tokens? Doesn't make sense either, because then the user has no incentive to use anything except 0/max thinking budgets.

RobinL• 1 year ago

Does anyone know how this pricing works? Supposing I have a classification prompt where I need the response to be a binary yes/no. I need one token of output, but reasoning will obviously add far more than 6 additional tokens. Is it still a 6x price multiplier? That doesn't seem to make sense, but not does paying 6x more for every token including reasoning ones

coder543• 1 year ago

"When you have thinking turned on, all output tokens (including thoughts) are charged at the $3.50 / 1M rate"[0]

[0] https://x.com/OfficialLoganK/status/1912981986085323231

punkpeye3• 1 year ago

This is cool, but rate limits on all of these preview models are PITA

Layvier3• 1 year ago

Agreed, it's not even possible to run an eval dataset. If someone from google see this please at least increase the burst rate limit

punkpeye3• 1 year ago

It is not without rate limits, but we do have elevated limits for our accounts through:

https://glama.ai/models/gemini-2.5-flash-preview-04-17

So if you just want to run evals, that should do it.

Though the first couple of days after a model comes out are usually pretty rough because everyone try to run their evals.

arnaudsm3• 1 year ago

Gemini flash models have the least hype, but in my experience in production have the best bang for the buck and multimodal tooling.

Google is silently winning the AI race.

belter• 1 year ago

Google is silently winning the AI race.

That is what we keep hearing here...The last Gemini I cancelled the account, and can't help notice the new one they are offering for free...

arnaudsm3• 1 year ago

Sorry I was talking of B2B APIs for my YC startup. Gemini is still far behind for consumers indeed.

Layvier3• 1 year ago

Absolutely. So many use cases for it, and it's so cheap/fast/reliable

danielbln3• 1 year ago

I want to use these almost too cheap to meter models like Flash more, what are some interesting use cases for those?

SparkyMcUnicorn• 1 year ago

And stellar OCR performance. Flash 2.0 is cheaper and more accurate than AWS Textract, Google Document AI, etc.

Not only in benchmarks[0], but in my own production usage.

[0] https://getomni.ai/ocr-benchmark

Fairburn• 1 year ago

Sorry, but no. Gemini isn't the fastest horse, yet. And it's use within their ecosystem means it isn't geared to the masses outside of their bubble. They are not leading the race but they are a contender.

spruce_tips3• 1 year ago

i have a high volume task i wrote an eval for and was pleasantly surprised at 2.0 flash's cost to value ratio especially compared to gpt4.1-mini/nano

accuracy | input price | output price

Gemini Flash 2.0 Lite: 67% | $0.075 | $0.30

Gemini Flash 2.0: 93% | $0.10 | $0.40

GPT-4.1-mini: 93% | $0.40 | $1.60

GPT-4.1-nano: 43% | $0.10 | $0.40

excited to to try out 2.5 flash

jay_kyburz3• 1 year ago

Can I ask a serious question. What task are you writing where its ok to get 7% error rate. I can't get my head around how this can be used.

BoredPositron• 1 year ago

The API is free, and it's great for everyday tasks. So yes there is no better bang for the buck.

drusepth• 1 year ago

Wait, the API is free? I thought you had to use their web interface for it to be free. How do you use the API for free?

statements2• 1 year ago

Absolutely agree. Granted, it is task dependent. But when it comes to classification and attribute extraction, I've been using 2.0 Flash with huge access across massive datasets. It would not be even viable cost wise with other models.

sethkim• 1 year ago

How "huge" are these datasets? Did you build your own tooling to accomplish this?

xnx6• 1 year ago

Shhhh. You're going to give away the secret weapon!

gambiting2• 1 year ago

In my experience they are as dumb as a bag of bricks. The other day I asked "can you edit a picture if I upload one"

And it replied "sure, here is a picture of a photo editing prompt:"

https://g.co/gemini/share/5e298e7d7613

It's like "baby's first AI". The only good thing about it is that it's free.

JFingleton2• 1 year ago

Prompt engineering is a thing.

Learning how to "speak llm" will give you great results. There's loads of online resources that will teach you. Think of it like learning a new API.

ghurtado2• 1 year ago

in my experience they are as dumb as a bag of bricks

In my experience, anyone that describes LLMs using terms of actual human intelligence is bound to struggle using the tool.

Sometimes I wonder if these people enjoy feeling "smarter" when the LLM fails to give them what they want.

nowittyusername• 1 year ago

Its because google hasn't realized the value of training the model on information about its own capabilities and metadata. My biggest pet peeve about google and the way they train these models.

rvz• 1 year ago

Google always has been winning the AI race as soon as DeepMind was properly put to use to develop their AI models, instead of the ones that built Bard (Google AI team).

GaggiX• 1 year ago

Flash models are really good even for an end user because how fast and good performance they have.

ghurtado2• 1 year ago

I know it's a single data point, but yesterday I showed it a diagram of my fairly complex micropython program, (including RP2 specific features, DMA and PIO) and it was able to describe in detail not just the structure of the program, but also exactly what it does and how it does it. This is before seeing a single like of code, just going by boxes and arrows.

The other AIs I have shown the same diagram to, have all struggled to make sense of it.

redbell• 1 year ago

Google is silently winning the AI race

Yep, I agree! This convinced me: https://news.ycombinator.com/item?id=43661235

ramesh314• 1 year ago

”Google is silently winning the AI race.”

It’s not surprising. What was surprising honestly was how they were caught off guard by OpenAI. It feels like in 2022 just about all the big players had a GPT-3 level system in the works internally, but SamA and co. knew they had a winning hand at the time, and just showed their cards first.

wkat42423• 1 year ago

True and their first mover advantage still works pretty well. Despite "ChatGPT" being a really uncool name in terms of marketing. People remember it because they were the first to wow them.

russellbeattie• 1 year ago

I have to say, I never doubted it would happen. They've been at the forefront of AI and ML for well over a decade. Their scientists were the authors of the "Attention is all you need" paper, among thousands of others. A Google Scholar search produces endless results. There just seemed to be a disconnect between the research and product areas of the company. I think they've got that worked out now.

They're getting their ass kicked in court though, which might be making them much less aggressive than they would be otherwise, or at least quieter about it.

Nihilartikel3• 1 year ago

100% agree. I had Gemini flash 2 chew through thousands of points of nasty unstructured client data and it did a 'better than human intern' level conversion into clean structured output for about $30 of API usage. I am sold. 2.5 pro experimental is a different league though for coding. I'm leveraging it for massive refactoring now and it is almost magical.

jdthedisciple2• 1 year ago

thousands of points of nasty unstructured client data

What I always wonder in these kinds of cases is: What makes you confident the AI actually did a good job since presumably you haven't looked at the thousands of client data yourself?

For all you know it made up 50% of the result.

cdelsolar• 1 year ago

what tool are you using 2.5-pro-exp through? Cline? Or the browser directly?

roygbiv23• 1 year ago

Isn't it better to get gemini to create a tool to format the data? Or was it in such a state that that would have been impossible?

tcgv• 1 year ago

I'm leveraging it for massive refactoring now and it is almost magical.

Can you share more about your strategy for "massive refactoring" with Gemini?

Like the steps in general for processing your codebase, and even your main goals for the refactoring.

no_wizard• 1 year ago

I remember everyone saying its a two horse race between Google and OpenAI, then DeepSeek happened.

Never count out the possibility of a dark horse competitor ripping the sod right out from under

nonethewiser2• 1 year ago

How is deepseak doing though? It seemed like they probably just ingested ChatGPT. https://www.forbes.com/sites/torconstantino/2025/03/03/deeps...

Still impressive but would really put a cap on expectations for them.

bhl• 1 year ago

It's cheap but also lazy. It sometimes generates empty strings or empty arrays for tool calls, and then I just re-route the request to a stronger model for the tool call.

I've spent a lot of time on prompts and tool-calls to get Flash models to reason and execute well. When I give the same context to stronger models like 4o or Gemini 2.5 Pro, it's able to get to the same answers in less steps but at higher token cost.

Which is to be expected: more guardrails for smaller, weaker models. But then it's a tradeoff; no easy way to pick which models to use.

Instead of SQL optimization, it's now model optimization.

paulcole• 1 year ago

Google is silently winning the AI race.

It’s not clear to me what either the “race” or “winning” is.

I use ChatGPT for 99% of my personal and professional use. I’ve just gotten used to the interface and quirks. It’s a good consumer product that I like to pay $20/month for and use. My work doesn’t require much in the way of monthly tokens but I just pay for the OpenAI API and use that.

Is that winning? Becoming the de facto “AI” tool for consumers?

Or is the race to become what’s used by developers inside of apps and software?

The race isn’t to have the best model (I don’t think) because it seems like the 3rd best model is very very good for many people’s uses.

transformi• 1 year ago

Bad day is going on google.

First the decleration of illegal monopoly..

and now... Google’s latest innovation: programmable overthinking.

With Gemini 2.5 Flash, you too can now set a thinking_budget—because nothing says "state-of-the-art AI" like manually capping how long it’s allowed to reason. Truly the dream: debugging a production outage at 2am wondering if your LLM didn’t answer correctly because you cheaped out on tokens. lol.

“Turn thinking off for better performance.” That’s not a model config, that’s a metaphor for Google’s entire AI strategy lately.

At this point, Gemini isn’t an AI product—it’s a latency-cost-quality compromise simulator with a text interface. Meanwhile, OpenAI and Anthropic are out here just… cooking the benchmarks

danielbln3• 1 year ago

Google's Gemini 2.5 pro model is incredibly strong, it's en par and at times better than Claude 3.7 in coding performance, being able to ingest entire videos into the context is something I haven seen elsewhere either. Google AI products have been anywhere between bad (Bard) to lackluster (Gemini 1.5), but 2.5 is a contender, in all dimensions. Google is also the only player that owns the entire stack, from research, software , data, compute hardware. I think they were slow to start but they've closed the gap since.

bsmith• 1 year ago

Using AI to debug code at 2am sounds like pure insanity.

mring336212• 1 year ago

the new normal

spiderice• 1 year ago

They're suggesting you'll be up at 2am debugging code because your AI code failed. Not that you'll be using AI to do the debugging.

hmaxwell• 1 year ago

I did some testing this morning:

Prompt: "can you find any mistakes on my codebase? I put one in there on purpose" + 70,000 tokens of codebase where in one line I have an include for a non-existent file.

Claude 3.7 Sonnet (Try 1 - new chat) Claimed it Found a race condition in the api of the admin interface that would be triggered if two admins were to change the room order at the same time. Claude suggested I group all sql queries in a single transaction. I looked at the code and found that it already used a transaction for all queries. I said: the order_update api is already done with a transaction. Claude replied: "You're absolutely right, and I apologize for my mistake. I was incorrect to claim there was a race condition issue. The transaction ensures atomicity and consistency of the updates, and the SQL queries are properly structured for their intended purpose."

Claude 3.7 Sonnet (Try 2 - new chat) Found a comment in the admin ui javascript code that suggested a potential alternative to event handler cleanup that was not implemented because I decided to go with a cleaner route. Then asked "Is this the issue you intentionally included, or would you like me to look for other potential problems?" I said: "The comment merely suggests an alternative, right?" claude said: "Yes, you're absolutely right. The comment is merely suggesting an alternative approach that isn't being used in the code, rather than indicating a mistake. So there's no actual bug or mistake in this part of the code - just documentation of different possible approaches. I apologize for misinterpreting this as an issue!"

Claude 3.7 Sonnet (Try 3 - new chat) When processing items out of the database to generate QR codes in the admin interface, Claude says that my code both attempts to generate QR codes with undefined data AS WELL AS saying that my error handling skips undefined data. Claude contradicts itself within 2 sentences. When asking about clarification Claude replies: Looking at the code more carefully, I see that the code actually has proper error handling. I incorrectly stated that it "still attempts to call generateQRCode()" in the first part of my analysis, which was wrong. The code properly handles the case when there's no data-room attribute.

Gemnini Advanced 2.5 Pro (Try 1 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase.

Gemnini Advanced 2.5 Pro (Try 2 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase.

Gemnini Advanced 2.5 Pro (Try 3 - new chat) Found the intentional error and said I should stop putting db creds/api keys into the codebase.

o4-mini-high and o4-mini and o3 and 4.5 and 4o - "The message you submitted was too long, please reload the conversation and submit something shorter."

Tiberium4• 1 year ago

The thread is about 2.5 Flash though, not 2.5 Pro. Maybe you can try again with 2.5 Flash specifically? Even though it's a small model.

dyauspitr3• 1 year ago

I don’t particularly care about the non frontier models though, I found the comment very useful.

airstrike• 1 year ago

Have you tried Claude Code?

danielbln3• 1 year ago

Those responses are very Claude, to. 3.7 has powered our agentic workflows for weeks, but I've been using almost only Gemini for the last week and feel the output is better generally. It's gotten much better at agentic workflows (using 2.0 in an agent setup was not working well at all) and I prefer its tuning over Clause's, more to the point and less meandering.

rendang• 1 year ago

3 different answers in 3 tries for Claude? Makes me curious how many times you'd get the same answer if you asked 10/20/100 times

bambax• 1 year ago

codebase where in one line I have an include for a non-existent file

Ok but you don't need AI for this; almost any IDE will issue a warning for that kind of error...

fandorin• 1 year ago

how did you put your whole codebase in a prompt for gemini?

Workaccount26• 1 year ago

OpenAI might win the college students but it looks like Google will lock in enterprise.

xnx6• 1 year ago

ChatGPT seems to have a name recognition / first-mover advantage with college students now, but is there any reason to think that will stick when today's high school students are using Gemini on their Chromebooks?

gundmc3• 1 year ago

Funny you should say that. Google just announced today that they are giving all college students one year of free Gemini advanced. I wonder how much that will actually move the needle among the youth.

Workaccount26• 1 year ago

My guess is that they will use it and still call it "ChatGPT"...

drob5186• 1 year ago

And every professor just groaned at the thought of having to read yet another AI-generated term paper.

anovick• 1 year ago

* Only in the U.S.

superfrank2• 1 year ago

Is there really lock in with AI models?

I built a product that uses and LLM and I got curious about the quality of the output from different models. It took me a weekend to go from just using OpenAI's API to having Gemini, Claude, and DeepSeek all as options and a lot of that time was research on what model from each provider that I wanted to use.

pydry• 1 year ago

For enterprise practically any SaaS gets used as one more thing to lock them into a platform they already have a relationship with (either AWS, GCP or Azure).

It's actually pretty dangerous for the industry to have this much vertical integration. Tech could end up like the car industry.

drob5186• 1 year ago

There isn’t much of a lock-in, and that’s part of the problem the industry is going to face. Everyone is spending gobs of money on training and if someone else creates a better one next week, the users can just swap it right in. We’re going to have another tech crash for AI companies, similar to what happened in 2001 for .coms. Some will be winners but they won’t all be.

ein0p4• 1 year ago

How will it lock in the enterprise if its market share of enterprise customers is half that of Azure (Azure also sells OpenAI inference, btw), and one third that of AWS?

kccqzy2• 1 year ago

The same reason why people enjoy BigQuery enough that their only use of GCP is BigQuery while they put their general compute spend on AWS.

In other words, I believe talking about cloud market share as a whole is misleading. One cloud could have one product that's so compelling that people use that one product even when they use other clouds for more commoditized products.

asadm• 1 year ago

funny thing about younglings, they will migrate to something else as fast as they came to you.

drob5186• 1 year ago

I read about that on Facebook.

Oras• 1 year ago

Enterprise has already been won by Microsoft (Azure), which runs on OpenAI.

r00fus• 1 year ago

That isn't what I'm seeing with my clientele (lots of startups and mature non-tech companies). Most are using Azure but very few have started to engage AI outside the periphery.

jimbob45• 1 year ago

Came to say this. No respectable CTO would ever push a Google product to their superiors knowing Google will kill it in 1-3 years and they’ll look foolish for having pushed it.

edaemon• 1 year ago

It seems more and more like AI is less of a product and more of a feature. Most people aren't going to care or even know about the model or the company who made it, they're just going to use the AI features built into the products they already use.

esafak2• 1 year ago

That's going to be true until we reach AGI, when there will be a qualitative difference and we will lose our ability to discern which is better since they're too far ahead of us.

statements2• 1 year ago

Interesting to note that this might be the only model with knowledge cut off as recent as 2025 January

Tiberium4• 1 year ago

Gemini 2.5 Pro has the same knowledge cutoff specified, but in reality on more niche topics it's still limited to ~middle of 2024.

brightball• 1 year ago

Isn't Grok 3 basically real time now?

Tiberium4• 1 year ago

That's the web version (which has tools like search plugged in), other models in their official frontends (Gemini on gemini.google.com, GPT/o models on chatgpt.com) are also "real time". But when served over API, most of those models are just static.

bearjaws• 1 year ago

No LLM is real time, and in fact, even a 2025 cut off isn't entirely realistic. Without guidance to say, a new version of a framework it will frequently "reference" documentation from old versions and use that.

It's somewhat real time when it searches the web, of course that data is getting populated into context rather than in training.

jiocrag2• 1 year ago

Not at all. The model weights and training data remain the same, it's just RAG'ing real-time twitter data into its context window when returning results. It's like a worse version of Perplexity.

ein0p4• 1 year ago

Absolutely decimated on metrics by o4-mini, straight out of the gate, and not even that much cheaper on output tokens (o4-mini's thinking can't be turned off IIRC).

gundmc3• 1 year ago

It's good to see some actual competition on this price range! A lot of Flash 2.5's edge will depend on how well the dynamic reasoning works. It's also helpful to have _significantly_ lower input token cost for a large context use cases.

rfw300• 1 year ago

o4-mini does look to be a better model, but this is actually a lot cheaper! It's ~7x cheaper for both input and output tokens.

ein0p4• 1 year ago

These small models only make sense with "thinking" enabled. And once you enable that, much of the cost advantage vanishes, for output tokens.

vessenes2• 1 year ago

o4-mini costs 8x as much as 2.5 flash. I believe its useful context window is also shorter, although I haven't verified this directly.

mccraveiro• 1 year ago

2.5 flash with reasoning is just 20% cheaper than o4-mini

mupuff1234• 1 year ago

Not sure "decimated" is a fitting word for "slightly higher performance on some benchmarks".

fwip• 1 year ago

Perhaps they were using the original meaning of "one-tenth destroyed." :P

ein0p4• 1 year ago

66.8% error rate reduction for o4-mini on AIME2025, and 21% error rate reduction on MMMU isn't "slightly higher". It'll be quite noticeable in practice.

kfajdsl• 1 year ago

Anecdotally o4-mini doesn’t perform as well on video understanding tasks in our pipeline, and also in Cursor it seems really not great.

During one session, it read the same file (same lines) several times, ran ‘python -c ‘print(“skip!”)’’ for no reason, and then got into another file reading loop. Then after asking a hypothetical about the potential performance implications of different ffmpeg flags, it claimed that it ran a test and determined conclusively that one particular set was faster, even though it hadn’t even attempted a tool call, let alone have the results from a test that didn’t exist.

xbmcuser4• 1 year ago

For a non programmer like me google is becoming shockingly good. It is giving working code the first time. I was playing around with it asked it to write code to scrape some data of a website to analyse. I was expecting it to write something that would scrape the data and later I would upload the data to it to analyse. But it actually wrote code that scraped and analysed the data. It was basic categorizing and counting of the data but I was not expecting it to do that.

kccqzy2• 1 year ago

That's the opposite experience of my wife who's in tech but also a non programmer. She wanted to ask Gemini to write code to do some basic data analysis things in a more automated way than Excel. More than once, Gemini wrote a long bash script where some sed invocations are just plain wrong. More than once I've had to debug Gemini-written bash scripts. As a programmer I knew how bash scripts aren't great for readability so I told my wife to ask Gemini to write Python. It resulted in higher code quality, but still contained bugs that are impossible for a non programmer to fix. Sometimes asking a follow up about the bugs would cause Gemini to fix it, but doing so repeatedly will result in Gemini forgetting what's being asked or simply throwing an internal error.

Currently IMO you have to be a programmer to use Gemini to write programs effectively.

sbarre• 1 year ago

I've found that good prompting isn't just about asking for results but also giving hints/advice/direction on how to go about the work.

I suspect that if Gemini is giving you bash scripts it's because you're note giving it enough direction. As you pointed out, telling it to use Python, or giving it more expectations about how to go about the work or how the output should be, will give better results.

When I am prompting for technical or data-driven work, I tend to almost walk through what I imagine the process would be, including steps, tools, etc...

xbmcuser4• 1 year ago

I had similar experiences few months back that is why I am saying it is becoming shockingly good the 2.5 is a lot better than the 2.0 version. Another thing I have realized just like google search in the past your query has a lot to do with the results you get. So an example of what you want works at getting better results

999900000999• 1 year ago

Let's hope that's the case for a while.

I want to be able to just tell chat GPT or whatever to create a full project for me, but I know the moment it can do that without any human intervention, I won't be able to find a job.

drob5186• 1 year ago

IMO, the only thing that’s consistent about AIs is how inconsistent they are. Sometimes, I ask them to write code and I’m shocked at how well it works. Other times, I feel like I’m trying to explain to a 5-year-old Alzheimer’s patient what I want and it just can’t seem to do the simplest stuff. And it’s the same AI in both cases.

SweetSoftPillow2• 1 year ago

It must have something to do with the way your wife is prompting. I've noticed this with my friends too. I usually get working code from Gemini 2.5 Pro on the first try, and with a couple of follow-up prompts, it often improves significantly, while my friends seem to struggle communicating their ideas to the AI and get worse results.

Good news: Prompting is a skill you can develop.

Workaccount26• 1 year ago

There is definitely an art to doing it, but the ability is definitely there even if you don't know the language at all.

I have a few programs now that are written in Python (2 by 3.7, one by 2.5) used for business daily, and I can tell you I didn't, and frankly couldn't, check a single line of code. One of them is ~500 LOC, the other two are 2200-2700 LOC.

yakz• 1 year ago

Ask it to write tests with the code and then ask it to fix the errors from the tests rather than just pointing out bugs. If you have an IDE that supports tool use (Claude Code, Roo Code) it can automate this process.

jiggawatts• 1 year ago

The AIs like many things out there work like an "evil genie". They'll give you what you asked for. The problem is typically that users ask for the wrong thing.

I've noticed beginners make mistakes like using singular terms when they should have used plural ("find the bug" vs "find the bugs"), or they fail to specify their preferred platform, language, or approach.

You mentioned your wife is using Excel, which is primarily used on Windows desktops and/or with the Microsoft ecosystem of products such as Power BI, PowerShell, Azure, SQL Server, etc...

Yet you mention she got a bash script using sed, both of which are from the Linux / GNU ecosystem. That implies that your wife didn't specify that she wanted a Microsoft-centric solution to her problem!

The correct answer here would have likely to have been to use Microsoft Fabric, which is an entire bag of data analysis and reporting tools that has data pipelines, automation, publishing, etc...

Or... just use the MashUp engine that's built-in to both Excel and PowerBI, which allows a surprisingly complex set of text, semi-structured, and tabular data processing. It can re-run the import and update graphs and charts with the new data.

PS: This is similar to going up to a Node.js programmer with a request. It doesn't matter what it is, they will recommend writing JavaScript to solve the problem. Similarly, a C++ developer will reach for C++ to solve everything they're asked to do. Right now, the AIs strongly prefer Linux, JavaScript, and especially Python for problem solving, because that's the bulk of the open-source code they were trained with.

dmos62• 1 year ago

Which Gemini was it? I've been using 2.5 Flash all day for programming ClojureScript via roo code and it's been great. Provided I'm using agent orchestration, a memory bank, and having it write docs for code it will work on.

ant6n• 1 year ago

Last time I tried Gemini, it messed with my google photo data plan and family sharing. I wish I could try the AI separate from my Google account.

jsnell7• 1 year ago

I wish I could try the AI separate from my Google account.

If that's a concern, just create another account. Doesn't even require using a separate browser profile, you can be logged into multiple accounts at once and use the account picker in the top right of most their apps to switch.

ModernMech4• 1 year ago

I've been continually disappointed. I've been told it's getting exponentially better and we won't be able to keep up with how good they get, but I'm not convinced. I'm using them every single day and I'm never shocked or awed by its competence, but instead continually vexxed that isn't not living up to the hype I keep reading.

Case in point: there was a post here recently about implementing a JS algorithm that highlighted headings as you scrolled (side note: can anyone remember what the title was? I can't find it again), but I wanted to test the LLM for that kind of task.

Pretty much no matter what I did, I couldn't get it to give me a solution that would highlight all of the titles down to the very last one.

I knew what the problem was, but even guiding the AI, it couldn't fix the code. I tried multiple AIs, different strategies. The best I could come up with was to guide it step by step on how to fix the code. Even telling it exactly what the problem was, it couldn't fix it.

So this goes out to the "you're prompting it wrong" crowd... Can you show me a prompt or a conversation that will get an AI to spit out working code for this task: JavaScript that will highlighting headings as you scroll, to the very last one. The challenge is to prompt it to do this without telling it how to implement it.

I figure this should be easy for the AI because this kind of thing is very standard, but maybe I'm just holding it wrong?

jsnell7• 1 year ago

Even as a human programmer I don't actually understand your description of the problem well enough to be confident I could correctly guess your intent.

What do you mean by "highlight as you scroll"? I guess you want a single heading highlighted at a time, and it should be somehow depending on the viewport. But even that is ambiguous. Do you want the topmost heading in the viewport? The bottom most? Depending on scroll direction?

This is what I got one-shot from Gemini 2.5 Pro, with my best guess at what you meant: https://gemini.google.com/share/d81c90ab0b9f

It seems pretty good. Handles scrolling via all possible ways, does the highlighting at load too so that the highlighting is in effect for the initial viewport too.

The prompt was "write me some javascript that higlights the topmost heading (h1, h2, etc) in the viewport as the document is scrolled in any way".

So I'm thinking your actual requirements are very different than what you actually wrote. That might explain why you did not have much luck with any LLMs.

croemer• 1 year ago

"Overengineered anchor links": https://news.ycombinator.com/item?id=43570324

__alexs• 1 year ago

Does billing for the API actually work properly yet?

alecco• 1 year ago

Gemini models are very good but in my experience they tend to overdo the problems. When I give it things for context and something to rework, Gemini often reworks the problem.

For software it is barely useful because you want small commits for specific fixes not a whole refactor/rewrite. I tried many prompts but it's hard. Even when I give it function signatures of the APIs the code I want to fix uses, Gemini rewrites the API functions.

If anybody knows a prompt hack to avoid this, I'm all ears. Meanwhile I'm staying with Claude Pro.

byearthithatius2• 1 year ago

Yes, it will add INSANE amounts of "robust error handling" to quick scripts where I can be confident about assumptions. This turns my clean 40 lines of Python where I KNOW the JSONL I am parsing is valid into 200+ lines filled with ten new try except statements. Even when I tell it not to do this, it loves to "find and help" in other ways. Quite annoying. But overall it is pretty dang good. It even spotted a bug I missed the other day in a big 400+ line complex data processing file.

zhengyi13• 1 year ago

I wonder how much of that sort of thing is driven by having trained their models on their own internal codebases? Because if that's the case, careful and defensive being the default would be unsurprising.

stavros• 1 year ago

I didn't realize this was a bigger trend, I asked it to write a simple testing script that POSTed a string to a local HTTP server as JSON, and it wrote a 40 line script, handling any possible error. I just wanted two lines.

dherikb• 1 year ago

I have the same issue using it with Aider.

The model is good to solve problems, but is very difficult to control the unnecessary changes that the model does in the rest of the code. Also it adds a lot of unnecessary comments, even when I explicitly say to not add.

For now Deepseek R1 and V3 it's working better to me, producing more predictable results and capturing better my intentions (not tried Claude yet).

w4yai• 1 year ago

Here's what I found to be working (not 100% but it gives much better and consistant results)

Basically, I ask it to repeat at the start of each message some rules :

"From now on, you must repeat and comply the following rules at the top of all your messages onwards:

- I will never rewrite API functions. Even if I think it's a good idea, it is a bad idea. I will keep the API function as it is and it is perfect like that.

- I will never add extra input validation. Even if I think it's a good idea, it is a bad idea. I will keep the function without validation and it is perfect like that.

- ...

- If I violate any of those rules, I did a bad job. "

Forcing it to repeat things make the model output more aligned and focused in my experience.

ks2048• 1 year ago

If this announcement is targeting people not up-to-date on the models available, I think they should say what "flash" means. Is there a "Gemini (non-flash)"?

I see the 4 Google model names in the chart here. Are these 4 the main "families" of models to choose from?

- Gemini-Pro-Preview

- Gemini-Flash-Preview

- Gemini-Flash

- Gemini-Flash-Lite

mwest217• 1 year ago

Gemini has had 4 families of models, in order of decreasing size:

- Ultra

- Pro

- Flash

- Flash-Lite

Versions with `-Preview` at the end haven't had their "official release" and are technically in some form of "early access" (though I'm not totally clear on exactly what that means given that they're fully available and as of 2.5 Pro Preview, have pricing attached to them - earlier versions were free during Preview but had pretty strict rate limiting but now it seems that Preview models are more or less fully usable).

drob5186• 1 year ago

Is GMail still in beta?

jsnell7• 1 year ago

The free-with-small-rate-limits designator was "experimental", not "preview".

I think the distinction between preview and full release is that the preview models have no guarantees on how long they'll be available, the full release comes with a pre-set discontinuation date. So if want the stability for a production app, you wouldn't want to use a preview model.

AStonesThrow• 1 year ago

I've been leveraging the services of 3 LLMs, mainly: Meta, Gemini, and Copilot.

It depends on what I'm asking. If I'm looking for answers in the realm of history or culture, religion, or I want something creative such as a cute limerick, or a song or dramatic script, I'll ask Copilot. Currently, Copilot has two modes: "Quick Answer"; or "Think Deeply", if you want to wait about 30 seconds for a good answer.

If I want info on a product, a business, an industry or a field of employment, or on education, technology, etc., I'll inquire of Gemini.

Both Copilot and Gemini have interactive voice conversation modes. Thankfully, they will also write a transcript of what we said. They also eagerly attempt to engage the user with further questions and followups, with open questions such as "so what's on your mind tonight?"

And if I want to know about pop stars, film actors, the social world or something related to tourism or recreation in general, I can ask Meta's AI through [Facebook] Messenger.

One thing I found to be extremely helpful and accurate was Gemini's tax advice. I mean, it was way better than human beings at the entry/poverty level. Commercial tax advisors, even when I'd paid for the Premium Deluxe Tax Software from the Biggest Name, they just went to Google stuff for me. I mean, they didn't even seem to know where stuff was on irs.gov. When I asked for a virtual or phone appointment, they were no-shows, with a litany of excuses. I visited 3 offices in person; the first two were closed, and the third one basically served Navajos living off the reservation.

So when I asked Gemini about tax information -- simple stuff like the terminology, definitions, categories of income, and things like that -- Gemini was perfectly capable of giving lucid answers. And citing its sources, so I could immediately go find the IRS.GOV publication and read it "from the horse's mouth".

Oftentimes I'll ask an LLM just to jog my memory or inform me of what specific terminology I should use. Like "Hey Gemini, what's the PDU for Ethernet called?" and when Gemini says it's a "frame" then I have that search term I can plug into Wikipedia for further research. Or, for an introduction or overview to topics I'm unfamiliar with.

LLMs are an important evolutionary step in the general-purpose "search engine" industry. One problem was, you see, that it was dangerous, annoying, or risky to go Googling around and click on all those tempting sites. Google knew this: the dot-com sites and all the SEO sites that surfaced to the top were traps, they were bait, they were sometimes legitimate scams. So the LLM providers are showing us that we can stay safe in a sandbox, without clicking external links, without coughing up information about our interests and setting cookies and revealing our IPv6 addresses: we can safely ask a local LLM, or an LLM in a trusted service provider, about whatever piques our fancy. And I am glad for this. I saw y'all complaining about how every search engine was worthless, and the Internet was clogged with blogspam, and there was no real information anymore. Well, perhaps LLMs, for now, are a safe space, a sandbox to play in, where I don't need to worry about drive-by-zero-click malware, or being inundated with Joomla ads, or popups. For now.

cynicalpeace• 1 year ago

1. The main transformative aspect of LLMs has been in writing code.

2. LLMs have had less transformative aspects in 2025 than we anticipated back in late 2022.

3. LLMs are unlikely to be very transformative to society, even as their intelligence increases, because intelligence is a minor changemaker in society. Bigger changemakers are motivation, courage, desire, taste, power, sex and hunger.

4. LLMs are unlikely to develop these more important traits because they are trained on text, not evolved in a rigamarole of ecological challenges.

charcircuit• 1 year ago

500 RPD for the free tier is good enough for my coding needs. Nice.

AbuAssar• 1 year ago

I noticed that OpenAI don't compare their models to third party models in their announcement posts, unlike google, meta and the others.

jskherman• 1 year ago

They're doing the Apple strategy. Less spotlight for other third parties, and less awareness how they're lagging behind so that those already ignorantly locked into OpenAI would not switch. But at this point why would anyone do that when switching costs are low?

mmaunder5• 1 year ago

More great innovation from Google. OpenAI have two major problems.

The first is Google's vertically integrated chip pipeline and deep supply chain and operational knowledge when it comes to creating AI chips and putting them into production. They have a massive cost advantage at every step. This translates into more free services, cheaper paid services, more capabilities due to more affordable compute, and far more growth.

Second problem is data starvation and the unfair advantage that social media has when it comes to a source of continually refreshed knowledge. Now that the foundational model providers have churned through the common crawl and are competing to consume things like video and whatever is left, new data is becoming increasingly valuable as a differentiator, and more importantly, as a provider of sustained value for years to come.

SamA has signaled both of these problems when he made noises about building a fab a while back and is more recently making noises about launching a social media platform off OpenAI. The smart money among his investors know these issues to be fundamental in deciding if OAI will succeed or not, and are asking the hard questions.

If the only answer for both is "we'll build it from scratch", OpenAI is in very big trouble. And it seems that that is the best answer that SamA can come up with. I continue to believe that OpenAI will be the Netscape of the AI revolution.

The win is Google's for the taking, if they can get out of their own way.

jbverschoor• 1 year ago

Except that they train their model even when you pay. So yeah.. I'd rather not use their "evil"

dayvigo• 1 year ago

Source?

mkl• 1 year ago

This is false: https://ai.google.dev/gemini-api/terms

Keyframe2• 1 year ago

Google has the data and has the hardware, not to mention software and infrastructure talent. Once this Bismarck turns around and it looks like it is, who can parry it for real? They have internet.zip and all the previous versions as well, they have youtube, email, search, books, traffic, maps and business on it, phones and habits around it, even the OG social network, the usenet. It's a sleeping giant starting to wake up and it's already causing commotion, let's see what it does when it drinks morning coffee.

kriro• 1 year ago

Agreed. One of Google's big advantages is the data access and integrations. They are also positioned really well for the "AI as entertainment" sector with youtube which will be huge (imo). They also have the knowledge in adtech and well injecting adds into AI is an obvious play. As is harvesting AI chat data.

Meta and Google are the long term players to watch as Meta also has similar access (Insta, FB, WhatsApp).

eastbound• 1 year ago

They have the Excel spreadsheets of all startups and businesses of the world (well 50/50 with Microsoft).

And Atlassian has all the project data.

whyenot2• 1 year ago

Another advantage that Google has is the deep integration of Gemini into Google Office products and Gmail. I was part of a pilot group and got to use a pre-release version and it's really powerful and not something that will be easy for OpenAI to match.

mmaunder5• 1 year ago

Agreed. Once they dial in the training for sheets it's going to be incredible. I'm already using notebooklm to upload finance PDFs, then having it generate tabular data and copypasta into sheets, but it's a garage solution compared to just telling it to create or update a sheet with parsed data from other sheets, PDFs, docs, etc.

And as far as gmail goes, I periodically try to ask it to unsubscribe from everything marketing related, and not from my own company, but it's not even close to being there. I think there will continue to be a gap in the market for more aggressive email integration with AI, given how useless email has become. I know A16Z has invested in a startup working on this. I doubt Gmail will integrate as deep as is possible, so the opportunity will remain.

Workaccount26• 1 year ago

I frankly am in doubt of future office products. In the last month I have ditched two separate excel productivity templates in favor of bespoke wrappers on sqlite databases, written by Claude and Gemini. Easier to use and probably 10x as fast.

You don't need a 50 function swiss army knife when your pocket can just generate the exact tool you need.

jdgoesmarching• 1 year ago

You say deep integration, yet there is still no way to send a Gemini Canvas to Docs without a lot of tedious copy-pasting and formatting because Docs still doesn’t actually support markdown. Gemini in Google Office in general has been a massive disappointment for all but the most simplistic of writing tasks.

They can have the most advanced infrastructure in the world, but it doesn’t mean much if Google continues its infamous floundering approach to product. But hey, 2.5 pro with Cline is pretty nice.

chucky_z• 1 year ago

I have access to this now and I want it to work so bad and it's just proper shit. Absolute rubbish.

They really, truly need to fix this integration. Gemini in Google Docs is barely acceptable, it doesn't work at all (for me) in Gmail, and I've not yet had it do anything other than error in Google Sheets.

zoogeny12• 1 year ago

If the battle was between Altman and Pichai I'd have my doubts.

But the battle is between Altman and Hassabis.

I recall some advice on investment from Buffett regarding how he invests in the management team.

mdp20215• 1 year ago

Could you please expand, on both your points?

sidibe• 1 year ago

Sorry but my eyes rolled to the back of my head with this one. This is between two teams with tons of smart contributors, but the difference is one is more flexible and able to take risks vs the other that has many times more researchers and the world's best and most mature infrastructure/tooling. Its not a CEO vs CEO battle

throwup238• 1 year ago

Nobody has really talked about what I think is an advantage just as powerful as the custom chips: Google Books. They already won a landmark fair use lawsuit against book publishers, digitized more books than anyone on earth, and used their Captcha service to crowdsource its OCR. They've got the best* legal cover and all of the best sources of human knowledge already there. Then Youtube for video.

The chips of course push them over the top. I don't know how much Deep Research is costing them but it's by far the best experience with AI I've had so far with a generous 20/day rate limit. At this point I must be using up at least 5-10 compute hours a day. Until about a week ago I had almost completely written off Google.

* For what it's worth, I don't know. IANAL

dynm• 1 year ago

The amount of text in books is surprisingly finite. My best estimate was that there are ~10¹³ tokens available in all books (https://dynomight.net/scaling/#scaling-data), which is less than frontier models are already being trained on. On the other hand, book tokens are probably much "better" than random internet tokens. Wikipedia for example seems to get much higher weight than other sources, and it's only ~3×10¹⁰ tokens.

paxys• 1 year ago

LibGen already exists, and all the top LLM publishers use it. I don't know if Google's own book index provides a big technical or legal advantage.

jofzar5• 1 year ago

Something that is not specifically called out but is also super relevant is actually the transcription of YouTube videos.

Every video is machine transcribed and stored and then for larger videos the author will often transcribed them themselves.

This is something they have already, it doesn't need any more "work" to get it vs a competitor.

jppittma• 1 year ago

I would think the biggest advantage is YouTube. There's a lot of modern content for analysis that's uncontaminated by LLMs.

peterjliu• 1 year ago

another advantage is people want the Google bot to crawl their pages, unlike most AI companies

mmaunder5• 1 year ago

This is an underrated comment. Yes it's a big advantage and probably a measurable pain point for Anthropic and OpenAI. In fact you could just do a 1% survey of robots.txt out there and get a reasonable picture. Maybe a fun project for an HN'er.

jiocrag2• 1 year ago

Excellent point. If they can figure out how to either remunerate or drive traffic to third parties in conjunction with this, it would be huge.

newfocogi• 1 year ago

This is right on. I work for a company with somewhat of a data moat and AI aspirations. We spend a lot of time blocking everyone's bots except for Google. We have people whose entire job is it to make it faster for Google to access our data. We exist because Google accesses our data. We can't not let them have it.

CobrastanJorji2• 1 year ago

Reddit was an interesting case here. They knew that they had particularly good AI training data, and they were able to hold it hostage from the Google crawler, which was an awfully high risk play given how important Google search results are to Reddit ads, but they likely knew that Reddit search results were also really important to Google. I would love to be able to watch those negotiations on each side; what a crazy high stakes negotiation that must've been.

stefan_• 1 year ago

I don't know man, for months now people keep telling me on HN how "Google is winning", yet no normal person I ever asked knows what the fuck "Gemini" is. I don't know what they are winning, it might be internet points for all I know.

Actually, some of the people polled recalled the Google AI efforts by their expert system recommending glue on pizza and smoking in pregnancy. It's a big joke.

mmaunder5• 1 year ago

Try uploading a bunch of PDF bank statements to notebooklm and ask it questions. Or the results of blood work. It's jaw dropping. e.g. uploaded 7 brokerage account statements as PDFs in a mess of formats and asked it to generate table summary data which it nailed, and then asked it to generate actual trades to go from current position to a new position in shortest path, and it nailed that too.

Biggest issue we have when using notebooklm is a lack of ambition when it comes to the questions we're asking. And the pro version supports up to 300 documements.

Hell, we uploaded the entire Euro Cyber Resilience Act and asked the same questions we were going to ask our big name legal firm, and it nailed every one.

But you actually make a fair point, which I'm seeing too and I find quite exciting. And it's that even among my early adopter and technology minded friends, adoption of the most powerful AI tools is very low. e.g. many of them don't even know that notebookLM exists. My interpretation on this is that it's VERY early days, which is suuuuuper exciting for us builders and innovators here on HN.

kube-system• 1 year ago

While there are some first-party B2C applications like chat front-ends built using LLMs, once mature, the end game is almost certainly that these are going to be B2B products integrated into other things. The future here goes a lot further than ChatGPT.

shmoogy• 1 year ago

That was ages ago.

Their new models excel at many things. Image editing, parsing PDFs, and coding are what I use it for. It's significantly cheaper than the closest competing models (Gemini 2.5 pro, and flash experimental with image generation).

Highly recommend testing against openai and anthropic models - you'll likely be pleasantly surprised.

labrador• 1 year ago

If the only answer for both is "we'll build it from scratch", OpenAI is in very big trouble

They could buy Google+ code from Google and resurrect it with OpenAI branding. Alternately they could partner with Bluesky

parsimo2010• 1 year ago

I don't think the issue is solving the technical implementation of a new social media platform. The issue is whether a new social media platform from OpenAI will deliver the kind of value that existing platforms deliver. If they promise investors that they'll get TikTok/Meta/YouTube levels of content+interaction (and all the data that comes with it), but deliver Mastodon levels, then they are in trouble.

onlyrealcuzzo4• 1 year ago

The smart money among his investors know these issues to be fundamental in deciding if OAI will succeed or not, and are asking the hard questions.

OpenAI has already succeeded.

If it ends up being a $100B company instead of a $10T company, that is success. By a very large margin.

It's hard to imagine a world in which OpenAI just goes bankrupt and ends up being worth nothing.

bdangubic• 1 year ago

it goes bankrupt when the cost of running the business outweights the earnings in the long run

samuel• 1 year ago

I can, and I would say it's a likely scenario, say 30%. If they don't have a significant edge over their competitors in the capabilities of their models, what's left? A money losing web app, and some API services that I'm sure aren't very profitable either. They can't compete with Google, Grok, Meta, MS, Amazon... They just can't.

They can end being the Altavista of this era.

dyauspitr3• 1 year ago

I haven’t heard this much positive sentiment about Google in a while. Making something freely available really turns public sentiment around.

mark_l_watson5• 1 year ago

Nice! Low price, even with reasoning enabled. I have been working on a short new book titled “Practical AI with Google: A Solo Knowledge Worker's Guide to Gemini, AI Studio, and LLM APIs” but with all of Google’s recent announcements it might not be a short book.

serjester• 1 year ago

Just ran it on one of our internal PDF (3 pages, medium difficulty) to json benchmarks:

gemini-flash-2.0: 60 ish% accuracy 6,250 pages per dollar

gemini-2.5-flash-preview (no thinking): 80 ish% accuracy 1,700 pages per dollar

gemini-2.5-flash-preview (with thinking): 80 ish% accuracy (not sure what's going on here) 350 pages per dollar

gemini-flash-2.5: 90 ish% accuracy 150 pages per dollar

I do wish they separated the thinking variant from the regular one - it's incredibly confusing when a model parameter dramatically impacts pricing.

ValveFan6969• 1 year ago

I have been having similar performance issues, I believe they intentionally made a worse model (Gemini 2.5) to get more money out of you. However, there is a way where you can make money off of Gemini 2.5.

If you set the thinking parameter lower and lower, you can make the model spew absolute nonsense for the first response. It costs 10 cents per input / output, and sometimes you get a response that was just so bad your clients will ask for more and more corrections.

mpalmer2• 1 year ago

Wow, what apps have you made so I know never to use them?

zoogeny12• 1 year ago

Google making Gemini 2.5 Pro (Experimental) free was a big deal. I haven't tried the more expensive OpenAI models so I can't even compare, only to the free models I have used of theirs in the past.

Gemini 2.5 Pro is so much of a step up (IME) that I've become sold on Google's models in general. It not only is smarter than me on most of the subjects I engage with it, it also isn't completely obsequious. The model pushes back on me rather than contorting itself to find a way to agree.

100% of my casual AI usage is now in Gemini and I look forward to asking it questions on deep topics because it consistently provides me with insight. I am building new tools with the mind to optimize my usage to increase it's value to me.

PerusingAround• 1 year ago

This comment is exactly my experience, I feel like as if I had wrote it myself.

cjohnson318• 1 year ago

Yeah, my wife pays for ChatGPT, but Gemini is fine enough for me.

qwertox3• 1 year ago

Just be aware that if you don't add a key (and set up billing) youre granting Google the right to train on your data. To have persons read them and decide how to use them for training.

dr_kiszonka• 1 year ago

I was a big fan of that model but it has been replaced in AI Studio by its preview version, which, by comparison, is pretty bad. I hope Google makes the release version much closer to the experimental one.

zoogeny12• 1 year ago

I can confirm the model name in Run Settings has been updated to "Gemini 2.5 Pro Preview ..." when it used to be "Gemini 2.5 Pro (Experimental) ...".

I cannot confirm if the quality is downgraded since I haven't had enough time with it. But if what you are saying is correct, I would be very sad. My big fear is the full-fat Gemini 2.5 Pro will be prohibitively expensive, but a dumbed down model (for the sake of cost) would also be saddening.

dieortin• 1 year ago

The preview version is exactly the same as the experimental one afaik

gundmc3• 1 year ago

The AI Studio product lead said on Twitter that it is exactly the same model just renamed for clarity when pricing was announced

jeeeb• 1 year ago

After comparing Gemini Pro and Claude Sonnet 3.7 coding answers side by side a few times, I decided to cancel my Anthropic subscription and just stick to Gemini.

wcarss• 1 year ago

Google has killed so many amazing businesses -- entire industries, even, by giving people something expensive for free until the competition dies, and then they enshittify hard.

It's cool to have access to it, but please be careful not to mistake corporate loss leaders for authentic products.

blueyes• 1 year ago

One of the main advantages Anthropic currently has over Google is the tooling that comes with Claude Code. It may not generate better code, and it has a lower complexity ceiling, but it can automatically find and search files, and figure out how to fix a syntax error fast.

mamp• 1 year ago

I've been using Gemini 2.5 and Claude 3.7 for Rust development and I have been very impressed with Claude, which wasn't the case for some architectural discussions where Gemini impressed with it's structure and scope. OpenAI 4.5 and o1 have been disappointing in both contexts.

Gemini doesn't seem to be as keen to agree with me so I find it makes small improvements where Claude and OpenAI will go along with initial suggestions until specifically asked to make improvements.

onlyrealcuzzo4• 1 year ago

Yes, IME, Anthropic seemed to be ahead of Google by a decent amount with Sonnet 3.5 vs 1.5 Pro.

However, Sonnet 3.7 seemed like a very small increase, whereas 2.5 Pro seemed like quite a leap.

Now, IME, Google seems to be comfortably ahead.

2.5 Pro is a little slow, though.

I'm not sure which model Google uses for the AI answers on search, but I find myself using Search for a lot of things I might ask Gemini (via 2.5 Pro) if it was as fast as Search's AI answers.

Graphon13• 1 year ago

Just curious, what tool do you use to interface with these LLMs? Cursor? or Aider? or...

jessep• 1 year ago

I have had a few epic refactoring failures with Gemini relative to Claude.

For example: I asked both to change a bunch of code into functions to pass into a `pipe` type function, and Gemini truly seemed to have no idea what it was supposed to do, and Claude just did it.

Maybe there was some user error or something, but after that I haven’t really used Gemini.

I’m curious if people are using Gemini and loving it are using it mostly for one-shotting, or if they’re working with it more closely like a pair programmer? I could buy that it could maybe be good at one but bad at the other?

sleiben• 1 year ago

Same here. Especially for native app development with swift I had way better results and just sticked with Gemini-2.5-*

yieldcrv• 1 year ago

I also cancelled my Anthropic yesterday, not because of Gemini but because it was the absolute worst time for Anthropic to limit their Pro plan to upsell their Max plan when there is so much competition out there

Manus.im also does code generation in a nice UI, but I’ll probably be using Gemini and Deepseek

No Moat strikes again

fsndz2• 1 year ago

More and more people are coming to the realisation that Google is actually winning at the model level right now.

zaphirplane• 1 year ago

What’s with the Google cheer squad in this thread, usually it’s Google lost its way and is evil.

Can’t be employees cause usually there is a disclaimer

MagicMoonlight• 1 year ago

I haven’t met a single person that uses Gemini. Companies are using Copilot and individuals are using ChatGPT.

Also, why would I want Google to spy on my AI usage? They’re evil.

orangesun• 1 year ago

Until they ditch it or make it ad-ridden like everything else.

m3kw9• 1 year ago

Using Claude code and Codex CLI and then Aider with Gemini 2.5 pro, Aider is much faster because you feed in the files instead of using tools to start doing all kinds of whole know what spending 10x the tokens. I tried a relatively simple refactor which needed around 7 files changed, only Aider with 2.5 got it and in the first shot. Where as both Codex and Claude code completely fumbled it

goshx• 1 year ago

Same here! It is borderline stubborn at times and I need to prove it wrong. Still, it is the best model to use with Cursor, in my experience.

teleforce2• 1 year ago

obsequious

Thanks for the new word, I have to look it up.

"obedient or attentive to an excessive or servile degree"

Apparently it means an AI that mindlessly follow your logic and instructions without reasoning and articulation is not good enough.

nemomarx• 1 year ago

I think here it's referring to a common problem where the AI agrees with your position too easily, and/or changes it's answer if you tell it the answer is wrong instantly (therefore providing no stable true answer if you asked it something about a fact)?

Also the slightly over cheery tone maybe.

zoogeny12• 1 year ago

It's a bit of a fancy way to say "yes man". Like in corporations or politics, if a leader surrounds themselves with "yes men".

A synonym would be sycophantic which would be "behaving or done in an obsequious way in order to gain advantage." The connotation is the other party misrepresents their own opinion in order to gain favor or avoid disapproval from someone of a higher status. Like when a subordinate tries to guess what their superior wants to hear instead of providing an unbiased response.

I think that accurately describes my experience with some LLMs due to heavy handed RLHF towards agreeableness.

In fact, I think obsequious is a better word since it doesn't have the cynical connotation of sycophant. LLMs don't have a motive and obsequious describes the behavior without specifying the intent.

tkgally• 1 year ago

Another useful word in this context is “sycophancy,” meaning excessive flattery or insincere agreement. Amanda Askell of Anthropic has used it to describe a trait they try to suppress in Claude:

https://youtube.com/watch?v=ugvHCXCOmm4&t=10286

sans_souse• 1 year ago

I wonder if anyone here will know this one; I learned the word "obsequious" over a decade ago while working the line of a restaurant. I used to listen to the 2p2 (2 plus 2) poker podcasts during prep and they had a regular feature with David Sklansky (iirc) giving tips, stories, advice etc. This particular one he simply gave the word "obsequious" and defined it later. I remember my sous chef and I were debating what it could mean and I guessed it right. I still can't remember what it had to do with poker, but that's besides the point.

Maybe I can locate it

UltraSane2• 1 year ago

I had a very interesting long debate/discussion with Gemini 2.5 Pro about the Synapse-Evolve bank debacle among other things. It really feels like debating a very knowledgeable and smart human.

rat99882• 1 year ago

You didn't have a debate, you just researched a question.

jofzar5• 1 year ago

My work doesn't have access to 2.5 pro and all these posts are just making me want it so much more.

I hate how slow things are sometimes.

basch• 1 year ago

Can’t you just go into aistudio with any free gmail account?

i_love_retros• 1 year ago

Why is it free / so cheap (I seem to be getting charged a few cents a day using it with aider so not free but still crazy cheap compared to sonnet)

brendanfinan• 1 year ago

we know how Google makes money

redox99• 1 year ago

I've had many disappointing results with gemini 2.5 pro. For general queries possibly involving search, chatgpt and grok work better for me.

For code, gemini is very buggy in cursor, so I use Claude 3.7. But it might be partly cursor's fault.

rgoulter2• 1 year ago

The 1 million token context window also means you can just copy/paste so much source code or log output.

crossroadsguy• 1 year ago

One difference, and imho that’s a big difference — you can’t use any of the Google’s chatbots/models without being logged in, unlike chatgpt.

casey2• 1 year ago

It's a big deal, but not in the way that you think. A race to the bottom is humanities best defense against fast takeoff.

instagraham• 1 year ago

obsequious is such a nice word for this context, only possible in the AI age.

i'd find the same word improper to describe human beings - other words like plaintive, obedient and compliant often do the job better and are less obscure.

here it feels like a word whose time has come.

_blk3• 1 year ago

Have you tried Grok 3? It's a bit verbose for my taste even when prompted to be brief but answers seem better/more researched and less opinionated. It's also more willing to answer questions where the other models block an answer.

fuzzylightbulb• 1 year ago

A lot of people don't want to patronize the businesses of an unabashed Nazi sympathizer. There are more important things in life than model output quality.

zoogeny12• 1 year ago

I have not tried any of the Grok models but that is probably because I am rarely on X.

I have to admit I have a bias where I think Google is "business" while Grok is for lols. But I should probably take the time to asses it since I would prefer to have an opinion based on experience rather than vibes.

MetaWhirledPeas2• 1 year ago

100% of my casual AI usage is now in Gemini and I look forward to asking it questions on deep topics because it consistently provides me with insight.

It's probably great for lots of things but it doesn't seem very good for recent news. I asked it about recent accusations around xAI and methane gas turbines and it had no clue what I was talking about. I asked the same question to Grok and it gave me all sorts of details.

ramesh314• 1 year ago

It's probably great for lots of things but it doesn't seem very good for recent news.

You are missing the point here. The LLM is just the “reasoning engine” for agents now. Its corpus of facts are meaningless, and shouldn’t really be relied upon for anything. But in conjunction with a tool calling agentic process, with access to the web, what you described is now trivially doable. Single shot LLM usage is not really anything anyone should be doing anymore.

arizen2• 1 year ago

This was my experience as well.

Gemini performing the best on coding tasks, while giving underwhelming responses on recent news.

While Grok was OK for coding tasks, but being linked to X, provided best response on recent events.

thebiggening• 1 year ago

Tremendous spam. Just wonderful. Comments too.

minimaxir2• 1 year ago

One hidden note from Gemini 2.5 Flash when diving deep into the documentation: for image inputs, not only can the model be instructed to generated 2D bounding boxes of relevant subjects, but it can also create segmentation masks! https://ai.google.dev/gemini-api/docs/image-understanding#se...

At this price point with the Flash model, creating segmentation masks is pretty nifty.

The segmentation masks are a bit of a galaxy brain implementation by generating a b64 string representing the mask: https://colab.research.google.com/github/google-gemini/cookb...

I am trying to test it in AI Studio but it sometimes errors out, likely because it tries to decode the b64 lol.

behnamoh• 1 year ago

Wait, did they just kill YOLO, at least for time-insensitive tasks?

minimaxir2• 1 year ago

YOLO is probably still cheaper if bounding boxes are your main goal. Good segmentation models that work for arbitrary labels, however, are much more expensive to set up and run, so this type of approach could be an interesting alternative depending on performance.

daemonologist2• 1 year ago

No, the speed of YOLO/DETR inference makes it cheap as well - probably at least five or six orders of magnitude cheaper.

  Edit: After some experimentation, Gemini also seems to not perform nearly as well as a purpose-tuned detection model.

It'll be interesting to test this capability and see how it evolves though. At some point you might be able use it as a "teacher" to generate training data for new tasks.

vunderba• 1 year ago

Well no. You can run/host YOLO which means not having to submit potentially sensitive information to a company that generates a large amount of revenue from targeted advertising.

daemonologist2• 1 year ago

Interestingly if you run this in Gemini (instead of AI Studio) you get:

    I am sorry, but I was unable to generate the segmentation masks for _ in the image due to an internal error with the tool required for this task.

(Not sure if that's a real or hallucinated error.)

ipsum2• 1 year ago

The performance is basically so bad it's unusable though, segmentation models and object detection models are still the best, for now.

msp262• 1 year ago

I've had mixed results with the bounding boxes even on 2.5 pro. On complex images where a lot of boxes need to be drawn they're in the general region but miss the exact location of objects.

simonw5• 1 year ago

This is SO cool. I built an interactive tool for trying this out (bring your own Gemini API key) here: https://tools.simonwillison.net/gemini-mask

More details plus a screenshot of the tool working here: https://simonwillison.net/2025/Apr/18/gemini-image-segmentat...

I vibe coded it using Claude and O3.

xnx6• 1 year ago

There is a starter app in AI Studio that demos this: https://aistudio.google.com/apps/bundled/spatial-understandi...

simonw5• 1 year ago

I spotted something interesting in the Python API library code:

https://github.com/googleapis/python-genai/blob/473bf4b6b5a6...

  class ThinkingConfig(_common.BaseModel):
      """The thinking features configuration."""
   
      include_thoughts: Optional[bool] = Field(
          default=None,
          description="""Indicates whether to include thoughts in the response. If true, thoughts are returned only if the model supports thought and thoughts are available.
        """,
      )
      thinking_budget: Optional[int] = Field(
          default=None,
          description="""Indicates the thinking budget in tokens.
          """,
      )

That thinking_budget thing is documented, but what's the deal with include_thoughts? It sounds like it's an option to have the API return the thought summary... but I can't figure out how to get it to work, and I've not found documentation or example code that uses it.

Anyone managed to get Gemini to spit out thought summaries in its API using this option?

phillypham• 1 year ago

They removed the docs and support for it https://github.com/googleapis/python-genai/commit/af3b339a9d....

You can see the thoughts in AI Studio UI as per https://ai.google.dev/gemini-api/docs/thinking#debugging-and....

lemming• 1 year ago

I maintain an alternative client which I build from the API definitions at https://github.com/googleapis/googleapis, which according to https://github.com/googleapis/python-genai/issues/345 should be the right place. But neither the AI Studio nor the Vertex definitions even have ThinkingConfig yet - very frustrating. In general it's amazing how much API munging is required to get a working client from the public API definitions.

qwertox3• 1 year ago

In AI Studio the flash moddels has two toggles: Enable thinking and Set thinking budget. If thinking budget is enabled, you can set tue max number of tokens it can use to think, else it's Auto.

Deathmax• 1 year ago

It is gated behind the GOOGLE_INTERNAL visibility flag, which only internal Google projects and Cursor have at the moment as far as I know.

msp262• 1 year ago

The API won't give you the "thinking" tokens, those are only visible on AI studio. Probably to try to stop distillation, very disappointing. I find reading the cot to be incredibly informative to identify failure modes.

Hey Everyone,
Moving forward, our team has made a decision to only show thoughts in Google AI Studio. Meaning, we no longer return thoughts via the Gemini API. Here is the updated doc to reflect that.

https://discuss.ai.google.dev/t/thoughts-are-missing-cot-not...

---

After I wrote all of that I see that the API docs page looks different today and now says:

Note that a summarized version of the thinking process is available through both the API and Google AI Studio.

https://ai.google.dev/gemini-api/docs/thinking

Maybe they just updated it? Or people aren't on the same page at Google idk

Previously it said

Models with thinking capabilities are available in Google AI Studio and through the Gemini API. Note that the thinking process is visible within Google AI Studio but is not provided as part of the API output.

https://web.archive.org/web/20250409174840/https://ai.google...

deanmoriarty• 1 year ago

Genuine naive question: when it comes to Google HN has generally a negative view of it (pick any random story on Chrome, ads, search, web, working at faang, etc. and this should be obvious from the comments), yet when it comes to AI there is a somewhat notable “cheering effect” for Google to win the AI race that goes beyond a conventional appreciation of a healthy competitive landscape, which may appear as a bit of a double standard.

Why is this? Is it because OpenAI is seen as such a negative player in this ecosystem that Google “gets a pass on this one”?

And bonus question: what do people think will happen to OpenAI if Google wins the race? Do you think they’ll literally just go bust?

antirez• 1 year ago

Maybe because Google is largely responsible, paying for the research, of most of the results we are seeing now. I'm not a Google fan, in the web side, and in their idea of what software engineering is, but they deserve to win the AI race, because right now all the other players provided a lot less than what Google did as public research. Also, with Gemini 2.5 PRO, there was a big hype moment, because the model is of unseen ability.

wkat42423• 1 year ago

Maybe they deserve it but it would be really bad for the world. Because they will enshittify the hell out of it once they're established. That's their MO.

I don't want Google to have a stranglehold over yet another type of online service. So I avoid them.

And things are going so fast now, whatever Google has today that might be better than the rest, in two months the rest will have it too. Of course Google will have something new again. But being 2 months behind isn't a huge deal. I don't have to have the 'winning' product. In fact most of my AI tasks go to an 8b llama 3.1 model. It's about on par with gpt 3.5 but that's fine.

01100011• 1 year ago

Didn't Google invent the transformer?

I think a lot of us see Google as both an evil advertiser and as an innovator. Google winning AI is sort of nostalgic for those of us who once cheered the "Do No Evil"(now mostly "Do Know Evil") company.

I also like how Google is making quiet progress while other companies take their latest incremental improvement and promote it as hard as they can.

pkaye• 1 year ago

I think for a while some people felt the Google AI models are worse but now its getting much better. On the other hand Google has their own hardware so they can drive down the costs of using the models so it keeps pressure on Open AI do remain cost competitive. Then you have Anthropic which has very good models but is very expensive. But I've heard they are working with Amazon to build a data center with Amazons custom AI chips so maybe they can bring down their costs. In the end all these companies will need a good model and lower cost hardware to succeed.

brap2• 1 year ago

I am cheering for the old Google to make a comeback and it seems like the AI race has genuinely sparked something positive inside Google.

wyre• 1 year ago

Gemini is just that good. From my usage it is much smarter than DeepSeek or Claude 3.7 Thinking models.

A lot of Google’s market share across its services comes from the monopoly effects Google has. The quality of Gemini 2.5 is noticeably smarter than its competitors so I see the applause for the quality of the LLM and not for Google.

I think it’s way too early to say anything about who is winning the race. There is still a long way to go; o3 scores highest in Humanity’s Last Exam (https://agi.safe.ai/) at 20%, 2.5 scores 18%.

sothatsit• 1 year ago

2.5 Pro is free, and I'm sure there's a lot of people who have just never tried the best models because they don't want to pay for them. So 2.5 Pro probably blows their socks off.

Whereas, if you've been paying for access to the best models from OpenAI and Anthropic all along, 2.5 Pro doesn't feel like such a drastic step-change. But going from free models to 2.5 Pro is a crazy difference. I also think this is why DeepSeek got so much attention so quickly - because it was free.

julianeon2• 1 year ago

It's been a while since they won something the "old" Google way: by building a superior product that is #1 on its merits.

In that sense Gemini is a throwback: there's no trick - it's objectively better than everything else.

sagarpatil• 1 year ago

Most of us weren’t using Gemini pro models (1.0, 1.5, 2.0) but the recent 2.5 pro is such a huge step up. It’s better than 3.7 sonnet for coding. Better than o1, o3-mini models and now o3 and o4-mini. It’s become my daily driver. It does everything I need with almost 100% accuracy, is cheap, fast, 1 million context window, uses google web search for grounding, can fetch YouTube video transcripts, can fetch website content, works in google workspace: Gmail, Docs, Sheets. Really hard to beat this combo. Oh and if you subscribe to their AI plan it comes with 2 TB drive storage.

oezi2• 1 year ago

The key is Gemini being free through AI Studio. This makes their technical improvement more impressive when OpenAI sells their best models at ridiculous prices.

If Google engages in price dumping as a monopolist remains to be seen but it feels like it.

The LLM race is fast paced and no moat has developed. People are switching on a whim if better models (by some margin) show up. When will OpenAI, Anthropic or DeepSeek counter 2.5 Pro? And will it be before Google releases the next Pro?

OpenAI commands a large chunk of the consumer market and they have considerable funds after their last round. They won't fold this or next year.

If Google wants to win this they must come up with a product strategy integrating their search business without seriously damaging their existing search business to much. This is hard.

int_19h8• 1 year ago

I dislike Google rather strongly due to their ad-based business model, and I was previously very skeptical of their AI offerings because of very lackluster performance compared to OpenAI and Claude. But I can't help but be impressed with Gemini Pro 2.5 for "deep research" and agentic coding. I have subscriptions with all three so that I can keep up with SOTA, but if I had to choose only one to keep, right now it'd be Gemini.

That said I still don't "cheer" for them and I would really rather someone else win the race. But that is orthogonal to recognition of observed objective superiority.

greentea23• 1 year ago

I prefer OpenAI and Anthropic big time because they are fresh players with less dominance over other aspects of digital life. Not having to login to an insidious tracker like Google is worth significantly worse performance. Although I have little FOMO here avoiding Gemini because evaluating these models on real world use cases remains quite subjective imo.

jonas21• 1 year ago

A lot of the negativity toward Google stems from the fact that they're the big, dominant player in search, ads, browsers, etc., rather than anything that they've done or any particular attribute of the company.

In AI, they're still seen as being behind OpenAI and others, so we don't see the same level of negativity.

summerlight3• 1 year ago

Because now it has brought real competitions to the field. GPT was the king and Claude had been the only meaningful challenger for a while but OpenAI didn't care about Anthropic but just be obsessed with Google. Gemini took a quite time to set the pipeline so initial version was not enough to push the frontier; you remember the days when Google released a new model, OpenAI just responded with some old models in their silo within a day only to crush them. That does not happen anymore and they're forced to develop a better model.

CephalopodMD• 1 year ago

As a googler working in LLM space, this feels like revisionist history to me haha! I remember a completely different environment only a few months ago when Anthropic was the darling child, and before that it was OpenAI (and for like 4 weeks somewhere in there, it was Deepseek). For literally years at this point, every time Bard or Gemini would make a major release, it would be largely ignored or put down in favor of the next "big thing" OpenAI was doing or Claude saturating coding benchmarks, never mind that Google was often just behind with the exact same tech ready to go, in some cases only missing their demo release by literally 1 day (remember live voice?). And every time this happened, folks would be posting things to the effect of "LOL I can't believe Google is losing the AI race - didn't they invent this?", "this is like Microsoft dropping the ball on mobile", "Google is getting their lunch eaten by scrappy upstarts," etc. I can't lie, it stings a bit when that's what you work on all day.

2.5 was quite good. Not stupidly good like the jump from GPT 2 to 3 or 3.5 to 4, but really good. It was a big jump in ELO and benchmarks. People like it, and I think it's just psychologically satisfying that the player everybody would have expected to win the AI race is currently in the lead. Gemini finally gets a day in the sun.

I'm sure this will change with whenever somebody comes up with the next big idea though. It probably won't take much to beat Gemini in the long run. There is literally zero moat.

krembo• 1 year ago

How is this sustainable for Google from business POV? It feels like Google is shooting itself in the foot while "winning" the AI race.. From my experience I think Google lost 99% of the ads it used to show me before in the search engine.

tomr75• 1 year ago

someone else will do it if they don't

aoeusnth1• 1 year ago

Their inference costs are the lowest in the business.

zenGull• 1 year ago

I've been paying for googles pro llm for about six months. At 20 it feels steep considering the free version is very good. I'm a devops work, and it's been very helpful. Ive tried gpt, copilot, Mixtral, Claude, etc and Geminis 1.5 pro was what sold me. The new 2.0 stuff is even better. Anecdotally, Gemini seems to forget to add stuff but doesn't hallucinate as much. I've been doing some pretty complex scripting this last week purely on Gemini fast 2.0 and it's been really really good.

jdthedisciple2• 1 year ago

Very excited to try it, but it is noteworthy that o4-mini is strictly better according to the very benchmarks shown by Google here.

Of course it's about 4x as expensive too (I believe), but still, given the release of openai/codex as well, o4-mini will remain a strong competitor for now.

thimabi3• 1 year ago

I find it baffling that Google offers such impressive models through the API and even the free AI Studio with fine-grained control, yet the models used in the Gemini app feel much worse.

Over the past few weeks, I’ve been using Gemini Advanced on my Workspace account. There, the models think for shorter times, provide shorter outputs, and even their context window is far from the advertised 1 million tokens. It makes me think that Google is intentionally limiting the Gemini app.

Perhaps the goal is to steer users toward the API or AI Studio, with the free tier that involves data collection for training purposes.

Alifatisk4• 1 year ago

Google lack marketing for ai studio, it has only recently become widely known through word of mouth

thimabi3• 1 year ago

That does work in Google’s favor. Users who are technical enough to want a better model eventually learn about AI Studio, while the rest are none the wiser.

_delirium• 1 year ago

This might have changed after you posted your comment, but it looks like 2.5 Pro and 2.5 Flash are available in the Gemini app now, both web and mobile.

thimabi3• 1 year ago

Oh, I didn’t mean to say that these models were unavailable through the app or website. Rather, I’ve realized that using them through the API or AI Studio yields much better results — even in the free tier.

You can check that by trying prompts with complex instructions and long inputs/outputs.

For instance, ask Gemini to generate notes from a specific source (say, a book or class transcription). Or ask it to translate a long article, full of idiomatic expressions, while maintaining high fidelity to the source. You will see that the very same Gemini models are underutilized on the app or the website, while their performance is stellar on the API or AI Studio.

bingdig• 1 year ago

It appears that this impacted gemini-2.5-pro-preview-03-25 somehow? grounding with google search no longer works.

I had a workflow running that would pull news articles from the past 24 hours. It now refuses to believe the current date is 2025-04-17. Even with search turned on and I ask it what the date is it and it always replies sometime in July 2024.

Alifatisk4• 1 year ago

No matter how good the new Gemini models have become, my bad experience with early Gemini is still stuck with me and I am afraid I still suffer from confirmation bias. Whenever I just look at the Gemini app, I already assume it’s going to be a bad experience.

thallavajhula• 1 year ago

At this point, at the current pace of AI model development, I feel like I can't tell which one is better. I usually end up using multiple LLMs to get a task done to my taste. They're all equally good and bad. It's like using GCP vs AWS vs Azure all over again, except in the AI space.

simonw5• 1 year ago

An often overlooked feature of the Gemini models is that they can write and execute Python code directly via their API.

My llm-gemini plugin supports that: https://github.com/simonw/llm-gemini

  uv tool install llm
  llm install llm-gemini
  llm keys set gemini
  # paste key here
  llm -m gemini-2.5-flash-preview-04-17 \
    -o code_excution 1 \
    'render a mandelbrot fractal in ascii art'

I ran that just now and got this: https://gist.github.com/simonw/cb431005c0e0535343d6977a7c470...

They don't charge anything extra for code execution, you just pay for input and output tokens. The above example used 10 input, 1,531 output which is $0.15/million for input and $3.50/million output for Gemini 2.5 Flash with thinking enabled, so 0.536 cents (just over half a cent) for this prompt.

blahgeek• 1 year ago

An often overlooked feature of the Gemini models is that they can write and execute Python code directly via their API.

Could you elaborate? I thought function calling is a common feature among models from different providers

WiSaGaN3• 1 year ago

This common feature requires the user of the API to implement the tool, in this case, the user is responsible to run the code the API outputs. The post you replied suggests that Gemini will run the code for the user behind the API call.

simonw5• 1 year ago

The Gemini API runs the Python code for you as part of your single API call, without you having to handle the tool call request yourself.

pantsforbirds• 1 year ago

See a example full in a few commands using uv think "wow I bet that Simon guy from twitter would love this" ... it's already him.

throaway920181• 1 year ago

I wish Gemini could do this with Go. It generates plenty of junk/non-parseable code and I have to feed it the error messages and hope it properly corrects it.

lleymrl651• 1 year ago

good

djrj477dhsnv• 1 year ago

Why are most comments here only comparing to Claude and just a few to ChatGPT and none to Grok?

Grok 3 has been my main LLM since its release. Is it not as good as I thought it was?

jofzar5• 1 year ago

IMO I will not use Grok while it's owned and related to Elon, not only do I not trust their privacy and data usage (not that I "really" trust open AI/Google etc) I just despise him.

It would have to be very significantly better for me to use it.

dyauspitr3• 1 year ago

Grok just isn’t the best out there.

WiSaGaN3• 1 year ago

Interesting that the output price per 1M tokens is $0.6 for non-reasoning, but $3.5 for reasoning. This seems to defy common assumption of how reasoning models work, and you tweak the <think> token probability to control how much thinking it does, but underlying it's the same model and the same inference code path.

michaelbrave• 1 year ago

Yesterday I started working through How to design programs, and set up a chat with Gemini 2.5 asking it to be my tutor as I go through it and to help answer my questions if I don't understand a part of the book. It has been knowledgeable, helpful and capable of breaking down complex things that I couldn't understand into understandable things. Fantastic all around.

zenkey• 1 year ago

Google is totally back in the game now, but it’s still going to take a lot more for them at this point to overcome OpenAI’s “first‑mover advantage” (clearly the favorite among younger users atm).

sweca5• 1 year ago

Google Pixel marketing is doing wonders for Gemini in young populations. I have been seeing a lot more of their phones in my generation's hands.

sinuhe69• 1 year ago

I’m not familiar with Python internals, so when I tried to convert a public AI model (not a LLM) to run locally, I got some problems no other AI could help. Asked Gemini 2.5 and it pin pointed the problem immediately. It solution was not practical but I guess it also works.

ashu1461• 1 year ago

One place where I feel gemini models lag is function calling and predicting correct arguments to function calls, is there a benchmark which scores models on the basis of this ?

upmind• 1 year ago

It's a shame that Gemini doesn't seem to have as much hype as GPT, I hope they gain more market share.

menshiki• 1 year ago

As a person mostly using AI for everyday tasks and business-related research, it's very impressive how quickly they've progressed. I would consider all models before 2.0 totally unusable. Their web interface, however, is so much worse than that of the ChatGPT macOS app.

gcbirzan• 1 year ago

Some aren't even at 2.0, and the version numbers aren't related in any way to their... generation? Also, what is so good about the ChatGPT app, specifically on macOS that makes it better?

convivialdingo2• 1 year ago

Dang - Google finally made a quality model that doesn’t make me want to throw my computer out a window. It’s honest, neutral and clearly not trained by the ideologically rabid anti-bias but actually super biased regime.

Did I miss a revolt or something in googley land? A Google model saying “free speech is valuable and diverse opinions are good” is frankly bizarre to see.

convivialdingo2• 1 year ago

Downvote me all you want - the fact remains that previous Google models were so riddled with guardrails and political correctness that it was practically impossible to use for anything besides code and clean business data. Random text and opinion would trigger a filter and shut down output.

Even this model criticizes the failures of the previous models.

tempaccount4202• 1 year ago

Yes, something definitely changed. It's still a little biased, it's kind of like OpenAI before Trump became president.

camkego• 1 year ago

The pricing table image in the article really should have included Gemini 2.5 pro. Sure, it could be after Flash to the right, but it would help people understand the price performance benefits of 2.5 Flash.

wanderr• 1 year ago

Gemini has the annoying habit of delegating tasks to me. Most recently I was trying to find out how to do something in FastRawViewer that I couldn't find a straightforward answer on. After hallucinating a bunch of settings and menus that don't exist, it told me to read the manual and check the user forums. So much for saving me time.

hubraumhugo2• 1 year ago

You can get your HN profile analyzed and roasted by it. It's pretty funny :) https://hn-wrapped.kadoa.com/

I'll add a selection for different models soon.

demaga• 1 year ago

Didn't expect to be roasted by AI this morning. Nice one

Alifatisk4• 1 year ago

How is this relevant to Gemini 2.5 Flash? I guess it's using it or something?

few• 1 year ago

This is cool.

Does it only use a few recent comments or entire history? I'm trying to figure out where it figured out my city when I thought I was careful not to reveal it. I'm scrolling back pages without finding where I said it in the past. Could it have inferred it based on other information or hallucinated it?

I wonder if there's a more opsec-focused version of this.

x187463• 1 year ago

Personal Projects

Will finally implement that gravity in TTE, despite vowing not to. We all know how well developers keep promises.

Knowledge Growth

Will achieve enlightenment on the true meaning of 'enshittification', likely after attempting to watch a single YouTube video without Premium.

I found these actually funny. Cool project.

CoolCold• 1 year ago

Error An error occurred in the Server Components render. The specific message is omitted in production builds to avoid leaking sensitive details. A digest property is included on this error instance which may provide additional details about the nature of the error.

hubraumhugo2• 1 year ago

fixed!

131hn• 1 year ago

If OpenAI offers Codex and Anthropic offers Claude Code, is there a CLI integration that Google recommends for using Gemini 2.5? That’s what’s keeping me, for now, with the other two.

yawaramin• 1 year ago

I just asked "why is 'Good Friday' so called?" and it got stuck. Flash 2.0 worked though.

uninformed-me00• 1 year ago

I want to think that this is all great, but the fact that this is also one of the best way to collect unsuspecting user data by default without explicit consent just doesn't feel right -- that applies to most people who would never have a chance reading this comment.

I don't want to be angry but screw these default opt-in to have your privacy violated free stuff.

Before you jump in to say you can pay to keep your privacy, stop and read again.

sgt• 1 year ago

How are they able to remain so competitive and will it last? The pricing almost seems too good to be true in terms of what they claim you get.

sweca5• 1 year ago

Custom TPUs ftw

egorfine• 1 year ago

I am always overlooking anything Google due to the fact that they are the opposite of "Don't be evil" and because their developer's console (Google Cloud) is incredibly hostile to humans.

Today I reluctantly clicked on their "AI Studio" link in the press-release and I was pleasantly surprised to discover that AI Studio has nothing in common with their typical UI/UX. It's nice and I love it!

brap2• 1 year ago

To be fair the UX of all GCP/AWS/Azure is ass. If you don’t know exactly what you’re looking for, good luck navigating that mess.

techwiz137• 1 year ago

I had a heart attack moment thinking they were bringing some form of Adobe Flash back.

latemedium• 1 year ago

There's an important difference between Gemini and Claude that I'm not sure how to quantify. I often use shell-connected LLMs (LLMs with a shell tool enabled) to take care of basic CSV munging / file-sorting tasks for me - I work in data science so there's a lot of this. When I ask Claude to do something, it carefully looks at all the directories and files before doing anything. Gemini, on the other hand, blindly jumps in and just starts moving stuff around. Claude executes more tools and is a little slower, but it almost always gets the right answer because it appropriately gathers the right context before really trying to solve the problem. Gemini doesn't seem to do this at all, but it makes a world of difference for my set of problems. Curious to see if others have had the same experience or if its just a quirk of my particular set of tasks

energy1233• 1 year ago

What's a shell connected LLM and how to do that?

kmacdough• 1 year ago

Look up Claude Code, Cursor, Aider and VSCode's agent integration. Generally, tools to use AI more actively for development. There are others as well. Plenty of info around. Here's not the place for a tutorial.

Alifatisk4• 1 year ago

Claude has always been the best at coding, no matter what all the benchmarks says, the people have spoken and the consensus is that Claude is the best.

rjurney• 1 year ago

I am building a knowledge graph using BAML [baml-py] to extract documents [it's opinionated towards docs] and then PySpark to ETL the data into a node / edge list. GPT4o got few relations... Gemini 2.5 got so many it was nuts, all accurate but not all from the article! I had to reign it in and instruct it not to build so vast a graph. Really cool, it knows a LOT about semiconductors :)

profsummergig• 1 year ago

I tried this prompt in both Gemini 2.5 Pro, and in ChatGPT.

"Draw me a timeline of all the dynasties of China. Imagine a horizontal line. Start from the leftmost point and draw segments for the start and end of each dynasty. For periods where multiple dynasties existed simultaneously draw parallel lines or boxes to represent the concurrent rule."

Gemini's response: "I'm just a language model, so I can't help you with that."

ChatGPT's response: an actual visual timeline.

renewiltord2• 1 year ago

All the communities where people think LLMs are junk love Gemini. Makes me sceptical that the enthusiasm is useful signal.

I found the full 2.0 useful for transcription of images. Very good OCR. But not a good assistant. Stalls often and once it has, loses context easily.

thegeomaster• 1 year ago

Is it possible that a community of people who are constantly pushing LLMs to their limits would be most aware of their limitations, and so more inclined to think they are junk?

In terms of business utility, Google has had great releases ever since the 2.0 family. Their models have never missed some mark --- either a good price/performance ratio, insane speeds, novel modalities (they still have the only API for autoregressive image generation atm), state-of-the-art long context support and coding ability (Gemini 2.5), etc.

However, most average users are using these models through a chat-like UI, or via generic tools like Cursor, which don't really optimize their pipelines to capture the strengths of different models. This way, it's very difficult to judge a model objectively. Just look at the obscene sycophancy exhibited by chatgpt-4o-latest and how it lifted LMArena scores.

ncr100• 1 year ago

Worked for me in 2.5 Flash, text only:

https://g.co/gemini/share/bcc257f9b0a0

asim• 1 year ago

I just wish the whole industry would stop using terms like thinking and reasoning. This is not what's happening. If we could come up with more appropriate terms that don't treat these models like they're human then we'd be in a much better place. That aside, it's cool to see the advancement of Google's offering.

dbbk• 1 year ago

Thinking perhaps, but why not reasoning?

lyu07282• 1 year ago

Do you think any machine will ever be able to think and/or reason? Or is that a uniquely human thing? and do you have a rational standard to judge when something is reasoning or thinking, or just vibes?

I'm asking because I wonder how much of that common attitude is just a sort of species-chauvinism. You are feeling anxious because machines getting smarter, you are feeling anger because "they" are taking your job away, but the machine doesn't do that, its people with an ideology that do that, you should be angry at that instead.

aerhardt• 1 year ago

I am only on OpenAI because they have a native Mac app. Call me old-school but my preferred workflow is still for the most part just asking narrow questions and copying-pasting back and forth. I've been playing with Junie (Jetbrain's AI agent) for a couple of days, but I still don't trust agents to run loose in my codebase for any sizeable amount of work.

Does anyone know if Google is planning native apps? Or any wrapping interfaces that work well on a Mac?

sweca5• 1 year ago

Raycast[0] has Gemini support in their AI offering and it's native, fast and intuitive.

[0] https://raycast.com/ai

sweca5• 1 year ago

Honestly, the best part about Gemini, especially as a consumer product, is their super lax, or lack thereof, ratelimits. They never have capacity issues, unlike Claude which always feels slow or sometimes outright rejects requests during peak hours. Gemini is constantly speedy and has extremely generous context window limits on the Gemini apps.

onlyrealcuzzo4• 1 year ago

Interesting. I use Claude quite a bit, and haven't encountered this.

Is this the free version of Claude or the paid version?

When are peak hours typically (in what timezone)?

sweca5• 1 year ago

I have Claude Pro and peak hours are in the afternoon and at night for me in EST

bossyTeacher2• 1 year ago

Is everyone on here solely evaluating the models on their programming capabilities? I understand this is HN but vibe coding LLM tools won't be able to sustain the LLM industry (let's not call it AI please)

barfingclouds• 1 year ago

I just need the Gemini app to allow push to talk :( Otherwise it’s not usable for me in the way I want it to be

piizeus2• 1 year ago

I don't think it is as good as deepseek v3 for updating coding tasks. I use 2.5 pro as architect with deepseek v3 as editor engineer. This combination work almost so perfect. Flash has advantages of great context size though. gpt 4.1 is also great.

Gemini 2.5 Flash

Related Stories