Two new Gemini models, reduced 1.5 Pro pricing, increased rate limits, and more

https://developers.googleblog.com/en/updated-production-ready-gemini-models-reduced-15-pro-pricing-increased-rate-limits-and-more/

by meetpateltech • 2 years ago

193 137 2 years ago

developers.googleblog.com

mixtureoftakes• 2 years ago

TLDR - 2x cheaper, slightly smarter, and they only compare those new models to their own old ones. Does google have moat?

usaar333• 2 years ago

The math score exceeds o1-preview (though not mini or o1 full) fwiw.

re-thc• 2 years ago

Does google have moat?

Potentially (depends if the EU cares)...

E.g. integration with Google search (instead of ChatGPT's Bing search), providing map data, android integration, etc...

ianbicking• 2 years ago

Their Android integration certainly isn't on track to earn them any moats... https://hachyderm.io/@ianbicking/113099247306589777

cj2• 2 years ago

Moat could be things like direct integration into Gmail (ask it to find your last 5 receipts from Amazon), Drive (chat with PDF), Slides (create images / flow charts), etc.

Not sure if their models are the moat. But they definitely have an opportunity from the productization perspective.

But so does Microsoft.

ethbr1• 2 years ago

Doesn't Microsoft also get OpenAI IP, if they run out of money?

svara3• 2 years ago

Have you tried the Gemini Gmail integration? I have that enabled in my GSuite account.

It's incredible how bad it is. I've seen it claim I've never received mail from a certain person, while the email was open right next to the chat widget. I've seen it tell me to use the standard search tool, when that wasn't suitable for the query. I've literally never had it find anything that wouldn't have been easier to find with the regular search.

I mean, it's a really obvious thing for them to do, I'm genuinely confused why they released it like that.

frankdenbow• 2 years ago

Interview with the product lead: https://x.com/rowancheung/status/1838611170061918575?

hadlock2• 2 years ago

They put a 42 minute video on twitter? That's brave.

mh-• 2 years ago

https://www.youtube.com/watch?v=WQvMdmk8IkM

throwup2382• 2 years ago

What makes the new model so unique?"

>> Yeah It's a good question. I think it's maybe less so of what makes it unique and more so the general trajectory of the trend that we're on.*

Disappointing.

summerlight• 2 years ago

Looks like they are more focused on the economical aspect of those large models? Like 90~95% performance of other frontier models at 50%~70% price.

TIPSIO• 2 years ago

I do like the trend.

Imagine if Anthropic or someone eventually release a Claude 3.5 but at like a whopping 10x its current speed.

Would be incredibly more useful and game changing than a slow o1 model that may or not be x percent smarter.

sgt101• 2 years ago

We might see that with the inference ASICs later this year I guess?

bangaladore• 2 years ago

Sonnet 3.5 is fast for its quality. But yeah, it's nowhere near Google's flash models. But I assume that is largely just because its a much smaller model.

Workaccount22• 2 years ago

They are going for large corporate customers. They are a brand name with deep pockets and a pretty risk-adverse model.

So even if Gemini sucks, they'll still win over execs being pushed to make a decision.

RhodesianHunter2• 2 years ago

That doesn't seem like much of a plan given their trailing position in the cloud space and the fact that Microsoft and AWS both have their own offerings.

hadlock2• 2 years ago

Not even trying to be snarky, but their lack of ability to offer products for more than a handful of years, does not lend google towards being chosen by large corporate customers. I know a guy who works in cloud sales and his government customers are PISSED they are sunsetting one of their PDF products and are being forced to migrate that process. The customer was expecting that to work for 10+ years and after a ~3 year onboarding process, they have 6 months to migrate. If my neck was on the line after buying the google PDF product, I wouldn't even short list them for an AI product.

simonw2• 2 years ago

This price drop is significant. For <128,000 tokens they're dropping from $3.50/million to $1.25/million, and output from $10.50/million to $2.50/million.

For comparison, GPT-4o is currently $5/million input and $15/million output and Claude 3.5 Sonnet is $3/million input and $15/million output.

Gemini 1.5 Pro was already the cheapest of the frontier models and now it's even cheaper.

GaggiX2• 2 years ago

GPT-4o is 2.5/10$. Unless you look at an old checkpoint. GPT-4o was the cheapest frontier model before.

simonw2• 2 years ago

I can’t see that price on https://openai.com/api/pricing/ - it’s listing $5/m input and $15/m output for GPT-4o right now.

No wait, correction: That’s confusing: it lists 4o first and then lists gpt-4o-2024-08-06 as $2.50/$10.

RhodesianHunter2• 2 years ago

I wonder if they're pulling the wall-mart model. Ruthlessly cut costs and sell at-or-below costs until your competitors go out of business, then ratchet up the prices once you have market dominance.

entropicdrifter• 2 years ago

You think Google would engage in monopolistic practices like that?

Because I do

bko• 2 years ago

Isn't Walmart still incredibly cheap? They have a net margin of 2.3%

I think that's one of those things competitors complain about that never actually happens (the raising prices part).

https://www.macrotrends.net/stocks/charts/WMT/walmart/net-pr...

lacker• 2 years ago

Probably not. Do they really believe they are going to knock OpenAI out of business, when the OpenAI models are better?

Instead I think they are going after the "Android model". Recognize they might not be able to dethrone the leader who invented the space. Define yourself in the marketplace as the cheaper alternative. "Less good but almost as good." In the end, they hope to be one of a small number of surviving members of an valuable oligopoly.

charlie02• 2 years ago

Yes, and it's the exact same thing OpenAI/Microsoft and Facebook are doing. In Facebook's case, they are giving it away for free.

sangnoir• 2 years ago

There's lot of room to cut margins in the AI stack right now (see Nvidia's latest report); low prices are not an sure indication of predatory pricing. Which company do you think is most likely to have the lowest training and inference costs between Anthropic, OpenAI and Google? My bet goes to the one designing,producing and using their own TPUs.

pzo• 2 years ago

whats confusing they have different pricing for output. Here [1] it's $5/million output (starting 1st october) and on vertex AI [2] it's $2.5/1 million (starting 7 october) - but characters - so it's overall gonna be more expensive if you wanna compare to equivalent 1 million tokens. It's actually even more confusing to know what kind of characters they mean? 1 byte? UTF-8?

[1] https://ai.google.dev/pricing

[2] https://cloud.google.com/vertex-ai/generative-ai/pricing

Deathmax3• 2 years ago

They do mention how characters are counted in the Vertex AI pricing docs: "Characters are counted by UTF-8 code points and white space is excluded from the count"

monsieurpooh2• 2 years ago

They CHANGED the pricing for the first link; originally the output said price drop from $10.50 to $2.50 but now it says $10.50 to $5.00.

lossolo• 2 years ago

For comparison, GPT-4o is currently $5/million input and $15/million output and Claude 3.5 Sonnet is $3/million input and $15/million output.

Google is the only one of the three that has its own data centers and custom inference hardware (TPU).

ekkk2• 2 years ago

It doesn't matter if it's cheap, it's unusable.

copperx• 2 years ago

Can you expound on why do you find it unusable?

monsieurpooh2• 2 years ago

They CHANGED the pricing from $2.50 to $5.00 stealthily unannounced. Look at the web site again; it says $5 per million now, and this comment on this website might be the ONLY evidence in the world that I wasn't gaslighting myself or hallucinating!

serjester• 2 years ago

Gemini feels like an abusive relationship — every few months, they announce something exciting, and I’m hopeful that this time will be different, that they’ve finally changed for the better, but every time, I’m left regretting having spent any time with them.

Their docs are awful, they have multiple unusable SDK's and the API is flaky.

For example, I started bumping into "Recitation" errors - ie they issue a flat out refusal if your response resembles anything in the training data. There's a GitHub issue with hundreds of upvotes and they still haven't published formal guidance on preventing this. Good luck trying to use the 1M context window.

Everything is built the "Google" way. It's genuinely unusable unless you're a total masochist and want to completely lock yourself into the Google ecosystem.

The only thing they can compete on is price.

jatins• 2 years ago

i think it's unusable if you are trying to use via GCP. Using via ai studio is a decent experience

thekevan2• 2 years ago

Google does not miss one single opportunity to miss an opportunity.

They announced a price reduction but it "won't be available for a few days". By the time, the initial hype will be over and the consumer-use side of the opportunity to get new users will be lost in other news.

phren0logy4• 2 years ago

As far as I can tell there's still no option for keeping data private?

diggan• 2 years ago

Makes sense, as soon as your data leaves your computer, it's safe to assume it's no longer private, no matter what promises a service gives you.

You want guaranteed private data that won't be used for anything? Keep it on your own computer.

9999000009993• 2 years ago

Not sure why your getting down voted. Anything sent to an cloud hosted LLM is subject to be publicly released or used in training.

Setting up a local LLM isn't that hard, although I'd probably air gap anything truly sensitive. I like ollama, but it wouldn't surprise me if it's phoning home.

sweca2• 2 years ago

If you are on the pay as you go model your data is exempted from training.

When you're using Paid Services, Google doesn't use your prompts (including associated system instructions, cached content, and files such as images, videos, or documents) or responses to improve our products, and will process your prompts and responses in accordance with the Data Processing Addendum for Products Where Google is a Data Processor. This data may be stored transiently or cached in any country in which Google or its agents maintain facilities.

https://ai.google.dev/gemini-api/terms

wildmXranat• 2 years ago

Interesting. In essence, we could equate to paying for the Gemini Advanced or Pro as a way to avoid use of our data and prompts.

https://ai.google.dev/gemini-api/terms#data-use-paid

kendallchuang4• 2 years ago

Has anyone used Gemini Code Assist? I'm curious how it compares with Github Copilot and Cursor.

spotlmnop• 2 years ago

God, it sucks

spiralk4• 2 years ago

The Aider leaderboards seem like a good practical test of coding usefulness: https://aider.chat/docs/leaderboards/. I haven't tried Cursor personally but I am finding Aider with Sonnet more useful that Github Copilot and its nice to be able to pick any model API. Eventually even a local model may be viable. This new Gemini model does not rank very high unfortunately.

kendallchuang4• 2 years ago

Thanks for the link. That's unfortunate, though perhaps the benchmarks will be updated after this latest Gemini release. Cursor with Sonnet is great, I'll have to give Aider a try as well.

dudus• 2 years ago

I use it and find it very helpful. Never tried cursor or copilot though

therein2• 2 years ago

I tried Cursor the other day. It was actually pretty cool. My thought was, I'll open this open source project and use it to grok my way around the codebase. It was very helpful. After that I accidentally pasted an API secret into the document; had to consider it compromised and re-issued the credential.

therein2• 2 years ago

I know you aren't necessarily talking about in-editor code assist but something about in-editor AI cloud code assist makes me super uncomfortable.

It makes sense I need to be careful not to commit secrets to public repositories but now I have to avoid not only saving credentials into a file but even to paste them by accident into my editor?

mil222• 2 years ago

I have used Github Copilot extensively within VS Code for several months. The autocomplete - fast and often surprisingly accurate - is very useful. My only complaint is when writing comments, I find the completions distracting to my thought process.

I tried Gemini Code Assist and it was so bad by comparison that I turned it off within literally minutes. Too slow and inaccurate.

I also tried Codestral via the Continue extension and found it also to be slower and less useful than Copilot.

So I still haven't found anything better for completion than Copilot. I find long completions, e.g. writing complete functions, less useful in general, and get the most benefit from short, fast, accurate completions that save me typing, without trying to go too far in terms of predicting what I'm going to write next. Fast is the key - I'm a 185 wpm on Monkeytype, so the completion had better be super low latency otherwise I'll already have typed what I want by the time the suggestion appears. Copilot wins on the speed front by far.

I've also tried pretty much everything out there for writing algorithms and doing larger code refactorings, and answering questions, and find myself using Continue with Claude Sonnet, or just Sonnet or o1-preview via their native web interfaces, most of the time.

imp0cat• 2 years ago

Have you tried Gitlab Duo and if so, what are your thoughts on that?

kendallchuang4• 2 years ago

I see, perhaps with Gemini because the model is larger it takes longer to generate the completions. I would expect with a larger model it would perform better on larger codebases. It sounds like for you, it's faster to work on a smaller model with shorter more accurate completions rather than letting the model guess what you're trying to write.

danmaz74• 2 years ago

I tried it briefly and didn't like it. On the other hand, I found Gemini pro better than sonnet or 4o at some more complex coding tasks (using continue.dev)

TheAceOfHearts• 2 years ago

I only use regular Gemini and the main feature I care about is absolutely terrible: summarizing YouTube videos. I'll ask for a breakdown or analysis of the video, and it'll give me a very high level overview. If I ask for timestamps or key points, it begins to hallucinate and make stuff up. It's incredibly disappointing that such a huge company with effectively unlimited access to both money and intellectual resources can't seem to implement a video analysis feature that doesn't suck. Part of me wonders if they're legitimately this incompetent or if they're deliberately not implementing good analysis features because it could eat into their views and advertisement opportunities.

foota• 2 years ago

I imagine you'd be paying more in ML costs than YouTube makes off your views.

bahmboo• 2 years ago

I've had it fail when a video does not have subtitles - I'm guessing that's what it uses. I have had good success having it answer the clickbait video titles like "is this the new best thing?"

It's not watching the video as far as I can tell.

msp26• 2 years ago

Has anyone tried Google's context caching feature? The minimum caching window being 32k tokens seems crazy to me.

jzebedee• 2 years ago

As someone who actually had to build on Gemini, it was so indefensibly broken that I couldn't believe Google really went to production with it. Model performance changes from day to day and production is completely unstable as Google will randomly decide to tweak things like safety filtering with no notice. It's also just plain buggy, as the agent scaffolding on top of Gemini will randomly fail or break their own internal parsing, generating garbage output for API consumers.

Trying to build an actual product on top of it was an exercise in futility. Docs are flatly wrong, supposed features are vaporware (discovery engine querying, anybody?), and support is nonexistent. The only thing Google came back with was throwing more vendors at us and promising that bug fixes were "coming soon".

With all the funded engagements and credits they've handed out, it's at the point where Google is paying us to use Gemini and it's _still_ not worth the money.

wewtyflakes• 2 years ago

Docs are flatly wrong

This +999; I couldn't believe how inconsistent and wrong the docs were. Not only that, but once I got something successfully integrated, it worked for a few weeks then the API was changed, so I was back to square one. I gave it a half-hearted try to fix it but ultimately said 'never again'! Their offering would have to be overwhelmingly better than Anthropic and OpenAI for me to consider using Gemini again.

victor106• 2 years ago

Same experience here.

I had hopes of Google able to compete with Claude and OpenAI. But I don’t think that’s the case. Unless they come out with a product that’s 10x better in the next year or so I think they lost the AI race.

ldjkfkdsjnv2• 2 years ago

The engineers at google are bad, they keep hiring via pure leetcode. Cant ship working products

naiv• 2 years ago

This sounds interesting:

"We will continue to offer a suite of safety filters that developers may apply to Google’s models. For the models released today, the filters will not be applied by default so that developers can determine the configuration best suited for their use case."

ancorevard2• 2 years ago

This is the most important update.

Pricing and speed doesn't matter when your call fails because of "safety".

CSMastermind2• 2 years ago

Also Google's safety filters are absolutely awful. Beyond parody levels of bad.

This is a query I did recently that got rejected for "safety" reasons:

Who are the current NFL starting QBs?

Controversial I know, I'm surprised I'd be willing to take the risk with submitting such a dangerous query to the model.

KaoruAoiShiho2• 2 years ago

There's still basic filters even if you take all the ones that you can turn off from the UI all off. It's still not capable of summarizing some YA novels I tried to feed it because of those filters.

panarky• 2 years ago

The "safety" filters used to make Gemini models nearly unusable.

For example, this prompt was apparently unsafe: "Summarize the conclusions of reputable econometric models that estimate the portion of import tariffs that are absorbed by the exporting nation or company, and what portion of import tariffs are passed through to the importing company or consumers in the importing nation. Distinguish between industrial commodities like steel and concrete from consumer products like apparel and electronics. Based on the evidence, estimate the portion of tariffs passed through to the importing company or nation for each type of product."

I can confirm that this prompt is no longer being filtered which is a huge win given these new lower token prices!

stan_kirdey• 2 years ago

Google should just offer llama3 405b, maybe slightly fine tuned. Geminis are unusable.

tiborsaas• 2 years ago

AI companies should not pick up naming models after astrological signs, after a while it will be hard to tell apart model reviews from horoscope.

dcchambers• 2 years ago

Geminis are unusable

how so?

JimDabell• 2 years ago

In my experience, Gemini models are far worse than any other frontier model when it comes to hallucinations. They are also pretty bad at getting caught in loops where pointing out a mistake makes it flap between two broken solutions. And obviously the overzealous softly stuff that other people have mentioned.

romland• 2 years ago

Pretty sure Google's got more than 700 million active users.

In fact, Google is most likely _the_ target for that clause in the license.

nmfisher• 2 years ago

I've found Gemini Pro to be the most reliable for function calling.

VirusNewbie• 2 years ago

Llama seems to be tricked up by simple puzzles that Gemini does not struggle with, in my experience.

Deathmax3• 2 years ago

They do: https://cloud.google.com/vertex-ai/generative-ai/docs/partne...

resource_waste• 2 years ago

Its just not as smart as ChatGPT or LLAMA, its mind boggling Google fell so far behind.

Mistletoe• 2 years ago

Has it changed much since March when this was written? Gemini won, 6-4. And it matches my experiences of just using Gemini instead of ChatGPT because it gives me more useful responses, when I feel like I want an AI answer.

https://www.tomsguide.com/ai/google-gemini-vs-openai-chatgpt

w23j• 2 years ago

"For this initial test I’ll be comparing the free version of ChatGPT to the free version of Google Gemini, that is GPT-3.5 to Gemini Pro 1.0."

The free version of ChatGPT is 4o now, isn't it? So maybe Gemini has not gotten worse, but the free alternatives are now better? When I compare ChatGPT-4o with Gemini-Advanced (wich is 1.5 Pro, I believe) the latter is just so much worse.

ldjkfkdsjnv2• 2 years ago

They have to drop the price because the model is bad. People will pay almost any cost for a model that is much better than the rest. How this company carries on the facade of competence is laughable. All the money on the planet, and they still cannot win on their core "competency".

FergusArgyll• 2 years ago

Any opinions on pro-002 vs pro-exp-0827 ?

Unlike others here I really appreciate the gemini API, it's free and it works. I haven't done too many complicated things with it but I made a chatbot for the terminal, a forecasting agent (for metaculus challenge) and a yt-dlp auto namer of songs. The point for me isn't really how it compares to openAI/anthropic, it's a free API key and I wouldn't have made the above if I had to pay just to play around

rkwasny• 2 years ago

Can someone explain to me why there is COMPLETELY different pricing for models on Vertex AI, Google AI studio and also OpenRouter has another price ...

hiddencost• 2 years ago

https://www.businessinsider.com/big-tech-org-charts-2011-6

Google is now looking more like the Microsoft chart.

sweca2• 2 years ago

No Human eval benchmark result?

kebsup• 2 years ago

Is there a good benchmark comparing multilingual and/or translation abilities of most recent LLMs? GPT-4o struggles for some tasks in my language learning app.

charlie02• 2 years ago

Anyone want to take bets on how long it takes for this to hit the Google Graveyard?

arresin• 2 years ago

I’ve used it. The API is incredibly buggy and flakey. A particular pain point is the “recitation error” fiasco. If you’re developing a real world app this basically makes the Gemini api unusable. It strikes me as a kind of “Potemkin” service.

Google is aware of the issue and it has been open on google's bug tracker since March 2024: https://issuetracker.google.com/issues/331677495

There is also discussion on GitHub: https://github.com/google-gemini/generative-ai-js/issues/138

It stems from something google added intentionally to prevent copyright material being returned verbatim (ala the NYT openai fiasco), so they dialled up the "recitation" control (the act of repeating training data—and maybe data they should not have legally trained on).

Here are some quotes from the bug tracker page:

I got this error by just asking "Who is Google?"
We're encountering recitation errors even with basic tutorials on application development. When bootstrapping a Spring Boot app, we're flagged for the pom.xml being too similar to some blog posts.
This error is a deal breaker... It occurs hundreds of times a day for our users and massively degrades their UX.

screye• 2 years ago

The recitation error is a big deal.

I was ready to champion gemini use across my organization, and the recitation issue curbed any enthusiasm I had. It's opaque and Google has yet to suggest a mitigation.

Your comment is not hyperbole. It's a genuine expression of how angry many customers are.

3choff• 2 years ago

It seems the recitation problem has been fixed with the latest models. In my tests, the answer generation no longer stops prematurely. Before I resume a project that was on hold due to this issue, I'm gathering feedback from other users about their experiences with this and the new models. Are you still having this problem?

anotherpaulg• 2 years ago

The new Gemini models perform basically the same as the previous versions on aider's code editing benchmark. The differences seem within the margin of error.

https://aider.chat/docs/leaderboards/

ramshanker• 2 years ago

One company buying expensive NVIDIA hardware vs another using in-house chips. Google got a huge advantage here. They could really undercut OpenAI.

rty32• 2 years ago

People have said that for many years. Very few companies are choosing Google's TPUs. Everyone wants H100s.

accumulator• 2 years ago

Cool, now all Google has to do is make it easier to onboard new GCP customers and more people will probably use it...its comical how hard it is to create a new GCP organization & billing account. Also I think more Workspace customers would probably try Gemini if it was a usage-based trial as opposed to clicking a "Try for 14 days" CTA to activate a new subscription.

pacoverdi• 2 years ago

Why the ** do they have to use a 8.9MiB png as hero image? I guess it was generated by AI?

mnicky• 2 years ago

I think the sentiment here is not fully objective, there are nice improvements in benchmarks (and even more so when accounting for the price): https://imgur.com/a/K3tVPEw

Also, this model shouldn't be compared to the CoT o1, I think. That is something different (also in price and speed).

Two new Gemini models, reduced 1.5 Pro pricing, increased rate limits, and more

Related Stories