Our next-generation model: Gemini 1.5

https://blog.google/technology/ai/google-gemini-next-generation-model-february-2024/

by todsacerdoti • 2 years ago

1244 588 2 years ago

crakenzak2• 2 years ago

Technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_...

The 1 million token context window + Gemini 1.0 Ultra level performance seems like it’ll unlock a wide range of incredible use cases!

HN, what are you going to use/build with this?

volkk• 2 years ago

was this posted by an AI bot

crakenzak2• 2 years ago

Lol nope I’m a normal person. Gimme a captcha and I’ll (hopefully) solve it ;)

mrkstu• 2 years ago

No, they're just applying their Twitter style engagement strategy to HN for some reason...

code512• 2 years ago

Dear Google, please fix your names and versioning.

Gemini Pro, Gemini Ultra... but was 1.0?

now upgraded but again Gemini Pro? jumping from 1.0 to 1.5?

wait but not Gemini Pro 1.5... Gemini "1.5" Pro

What actually happened between 1.0 and 1.5?

lairv2• 2 years ago

This naming is terrible, if I understand correctly this is the release of Gemini 1.5 Pro, but not Gemini 1.5 Ultra right ?

goalonetwo• 2 years ago

Looks like the former PM of chat at google found a new job.

cchance9• 2 years ago

How is that hard to understand? Yes its gemini 1.5 pro, they haven't released ultra or nano, like this isn't rocket science they didnt introduce Gemini 1.5 ProLight or something, lol its the Pro size model's 1.5 version.

jjbinx007• 2 years ago

They can't decide on a single name for a chat application so I think expecting them to come up with a sensible naming suggestion is optimistic at best.

aqme28• 2 years ago

Furthermore, is a minor version upgrade two months later really "next generation"?

philote• 2 years ago

Well if it's from 1 to 1.5 then it's really 5 minor version upgrades at once. And since 1.5 is halfway to 2 and you round up, it's next generation!

AndroTux• 2 years ago

Maybe it's not a "next generation" model, but rather their next model for text generation ;)

nkozyra• 2 years ago

Their inability to name things sensibly has been called out for years and it doesn't look like they care?

I'm not sure what the deal is, it has to be a marketing hinderance as every major tech company is trying to claw their way up the AI service mountain. Seems like the first step would be cogent naming.

data-ottawa• 2 years ago

It would have been better as Gemini Lite, Gemini, Gemini Pro, and then v1, v1.5 for model bumps.

Ultra vs pro vs nano with Ultra unlocked by buying Gemini Advanced is confusing.

I'm also not sure why they make base Gemini available after you have Advanced, because presumably there's no reason to use a worse model.

Alifatisk12• 2 years ago

I understod the transition as following.

Google Bard to Google Gemini is what they call Gemini 1.0.

Gemini consists of Gemini Nano, Gemini Pro, & Gemini Ultra.

Gemini Nano is for embedded and portable devices I guess? The free version of Gemini (gemini.google.com) is Gemini Pro. The paid version, called Gemini Advanced is using Gemini Ultra.

What we're reading now is about Gemini Pro version 1.0 switching to version 1.5 as of today.

meowface3• 2 years ago

That just made my head spin even more. (Like, I get it, but it's just a very tortuous naming system.) The free version is called Pro, Gemini Advanced is actually Gemini Ultra, the less powerful version upgraded to the more powerful model but the more powerful version is on the less powerful model.

People make fun of OpenAI for not using product names and just calling it "GPT" but at least it's straightforward: 2, 3, 3.5, 4. (On the API side it's a little more complicated since there's "turbo" and "instruct" but that isn't exposed to users, and turbo is basically the default.)

code512• 2 years ago

but if Vertex AI is using Gemini Ultra, then why makersuite (aisuite now? hmmm) showing only "Gemini 1.0 Pro 001" (001: a version inside a version)

and why have makersuite/aisuite in the first place, if Vertex AI is the center for all things AI? and why aitestkitchen?

I'm seeing only Gemini 1.0 Pro on Vertex AI. So even if I enabled Google Gemini Advanced (Ultra?), enabled Vertex AI API access, I have to first be blessed by Google to access advanced APIs.

It seems paying for their service doesn't mean anything to Google at this point. As a developer, you have to jump through hoops first.

growt• 2 years ago

It was probably not a wise choise to give the model itself and the product the same name: "Gemini Advanced is using Gemini Ultra". Also: "The free version ... is Gemini Pro" - is not what you usually see out there.

sho_hn13• 2 years ago

It's not that difficult.

Their LLM brand is now Gemini. Gemini comes in three different sizes, Nano/Pro/Ultra.

They recently released 1.0 versions of each, most recently (a few months after Nano and Pro) Ultra.

Today they are introducing version 1.5, starting with the Pro size. They say 1.5 Pro offers comparable performance to 1.0 Ultra, along with new abilities (token window size).

(I agree Small/Medium/Large would be better.)

apwell232• 2 years ago

What you described is difficult.

Alifatisk12• 2 years ago

They should remove the name Gemini Advanced and just stick to one name

OJFord3• 2 years ago

, starting with the Pro size

This is where it gets confusing IMO.

It's like if Apple announced macOS Blabahee, starting with Mini, not long after releasing Pro and Air touting benefits of Sonoma.

Also, just.. this is how TFA begins:

Last week, we rolled out our most capable model, Gemini 1.0 Ultra, [...] Our teams continue pushing the frontiers of our latest models with safety at the core. They are making rapid progress. [...] 1.5 Pro achieves comparable quality to 1.0 Ultra

Last week! And now we have next generation. And the wow is that it's comparable to the best of the previous generation. Ok fine at a smaller size, but also that's all we get anyway. Oh and the most capable remains the last generation one. As long as it's the biggest one.

Keyframe5• 2 years ago

What's Advanced then, chat? Also, by that, 1.5 Ultra is then still to come and it'll show even bigger guns.

mvkel• 2 years ago

So there's Nano 1.0, Pro 1.5, Ultra 1.0, but Pro 1.5 can only be accessed if you're a Vertex AI user (wtf is Vertex)?

That's very difficult.

pentagrama2• 2 years ago

Thank you, is more clear to me now. But I also read in some Google announcement about "Gemini Advanced", do you know what is that and the relation with the Nano/Pro/Ultra levels?

screye• 2 years ago

Gemini ultra 1.0 never went GA. So it is wierd that they'd release 1.5 when most can't even get their hands on 1.0 ultra.

mrcwinn• 2 years ago

I’m sure they had a discussion that size-based identifiers might imply models are primarily differentiated on the amount of knowledge they have. From that standpoint I don’t agree S/M/L would have been better.

gmuslera2• 2 years ago

Maybe they should take a hint on Windows versions name scheme and call the next version Gemini Meh.

apapapa• 2 years ago

Are you talking about Xbox one?

kccqzy3• 2 years ago

Dear OpenAI please fix your names and versioning. Why do you have GPT-3 and GPT-3.5? What happened between 3 and 3.5? And why isn't GPT-3 a single model? Why are there variations like GPT-3-6.7B and GPT-3-175b? And why is there now a turbo version? How does turbo compared to 4? And what's the relationship between the end-user product ChatGPT and a specific GPT model?

You see this problem isn't unique to Google.

lordswork2• 2 years ago

See https://news.ycombinator.com/item?id=39385230

cchance9• 2 years ago

This just means we'll be getting a Nano 1.5 and Ultra 1.5

and if Pro 1.5 is this good holy shit what will Ultra be...

Nano/Pro/Ultra are the model sizes, 1.0 or 1.5 is the version

hamburga• 2 years ago

“One of the key differentiators of this model is its incredibly long context capabilities, supporting millions of tokens of multimodal input. The multimodal capabilities of the model means you can interact in sophisticated ways with entire books, very long document collections, codebases of hundreds of thousands of lines across hundreds of files, full movies, entire podcast series, and more.”

skywhopper• 2 years ago

This is nice, but it’s hard to judge how nice without knowing more about how much compute and memory is involved in that level of processing. Obviously Google isn’t going to tell us, but without having some idea it’s impossible to judge whether this is an economically sustainable technology on which to start building dependencies in my own business.

criddell• 2 years ago

Sustainable? The countdown to cancellation on this project is already underway.

"Does it make sense today?" is really the only question you can ask and then build dependencies with the understanding that the entire thing will go away in 3-7 years.

freediver• 2 years ago

It would do Google a lot of service if every such announcement is not met with 'join the waitlist' and 'talk to your vertex ai team'.

baq• 2 years ago

Yeah compared to e.g. Apple’s ‘here’s our new iWidget 42 pro, you can buy it now’ it’s at best disappointing.

apozem• 2 years ago

Apple is good about only announcing real products you can buy. They don't do tech demos. It's always, "here's a problem. the new apple watch solves it. here're five other things the watch does. $399."

brianjking• 2 years ago

100%, I can't even use Imagen despite being an early tester of Vertex.

belval2• 2 years ago

They can't do that because only they are the incorruptible stewards empowered with the ability to develop these models, making them accessible to the unwashed masses would be irresponsible!

ethanbond• 2 years ago

The victim complex on this topic is getting really old.

They’re an enterprise software company doing an enterprise sales motion.

stavros2• 2 years ago

I'm generally an excited early adopter, but this kills my excitement immediately. I don't know if Gemini is out (or which Gemini is out) because I've associated Google with "you can't try their stuff", so I've learned to just ignore everything about Gemini.

hbn• 2 years ago

Google is really good at diluting any possible anticipation hardcore users might have for new stuff they do. 10 years ago I loved when there was a big update to one of their Android apps and I could sideload the apk from the internet to try it out early. Then they made all those changes A/B tests controlled by server side flags that would randomly turn themselves on and off, and there was no way to opt in or out. That was one of the (many) moves that contributed to my becoming disenchanted with Android.

petre2• 2 years ago

There is a Gemini service that you can use with your Google account, but it's kind of meh as it repeats your input, makes all sorts of assumptions. I am confused as well about the version. There's a link to another premium version (1.5?) on its page, to which I don't have access to without completing a quest which likely ends with a credit card input. That kills it for me.

skybrian• 2 years ago

I think the way to understand this is to realize that this isn’t targeted at a Hacker News audience and they don’t care what we think. The world doesn’t revolve around us.

What’s the goal? Maybe, being able to work with partners without it being a secret project that will inevitably leak, resulting in inaccurate stories in the press. What are non-goals? Driving sales or creating anticipation with a mass audience, like a movie trailer or an Apple product launch.

So they have to announce something, but most people don’t read Hacker News and won’t even hear about it until later, and that’s fine with them.

jpeter3• 2 years ago

and not having to wait months if you live in EU

lxgr3• 2 years ago

What's worse is that I can't seem to find a way to let Google know where I actually live (as opposed to where I am temporarily traveling, what country my currently inserted SIM card is from etc). And apparently there is no way to do this at all without owning an Android device!

Apple at least lets me change this by moving my iTunes/App Store account, which is its own ordeal and far from ideal, but at least there's a defined process: Tell us where you think you live, provide a form of payment from that place, maybe we'll believe you.

hobofan• 2 years ago

Eh, I think it's about as bad as the OpenAI method of officially announcing something and then "continuously rolling it out to all subscribers" which may be anything between a few days and months.

addandsubtract• 2 years ago

Remember when Gmail was new and you needed an invite to join? I guess Google is stuck in 2004.

bobchadwick• 2 years ago

I'm embarrassed to admit that I bought a Gmail invite on eBay for $6 when it was still invite-only.

moffkalast• 2 years ago

They don't seem to remember when that literally sunk Google+ because people had no use for a social network without their friends on it.

bachmeier• 2 years ago

This is bad practice across the board IMO. There seems to be an idea that this builds anticipation for new products. Sounds good in a PowerPoint presentation by an MBA but doesn't work in practice. Six months (or more!) after joining a waitlist, I'm not seeing it for the first time, so I don't really care when yet another email selling me something hits my inbox. I may not even open the email. This could be mitigated somewhat by at least offering a demo, but that's rare.

bushbaba2• 2 years ago

Likely they have limited capacity and are alloting things for highest paying and strategic customers

justrealist• 2 years ago

One PM in 2005 knocked it out of the park with Gmail and every Google PM since then has cargo-culted it.

kkzz99• 2 years ago

Its because they don't want you to actually use it and see how far behind they are compared to other companies. These announcement are meant to placate investors. "See, we are doing a lot of SotA AI too".

Keyframe5• 2 years ago

You might be right, but other things from Google tell the same story. For example, I recently tried to get ahold of Pixel 8 Pro. Had to import one from UK, and when I did, turns out that new feature of using thermometer on humans isn't available outside of USA. It doesn't even seem that process to certificate it outside of USA is in play. Google and sales/support just aren't a thing like with Apple, as a contrast. Which is a total shame. I know Google is strong, if not strongest in the game of tech, they just need to get their act together and I believe in them succeeding in that, but sales and support was never in their DNA. Not sure if that can be changed.

I'm more than happy to transfer my monthly $20 to google from OpenAI, on top of my youtube and google one subscription. It's up to Google to take it.

quatrefoil• 2 years ago

It lets the company control the narrative, without the distraction of fifty tech bloggers test-driving it and posting divergent opinions or findings. Instead, the conversation is anchored to what the company claims about the product.

It's interesting that it's the opposite of the gaming industry. There, because the reviewers dictate the narrative, the industry is better at ferreting out bogus claims. On the flip side, loud voices sometimes steamroll over decent products because of some ideological vendetta.

anonzzzies• 2 years ago

And region based. Yawn.

mil22• 2 years ago

Totally agree with this. I can see the desire to show off, but I don't understand how anyone can believe this is good marketing strategy. Any initial excitement I get from reading such announcements will be immediately extinguished when I discover I can't use the product yet. The primary impression I receive of the product is "vaporware." By the time it does get released I'll already have forgotten the details of the announcement, lost enthusiasm, and invested my time in a different product. When I'm choosing between AI services, I'll be thinking "no, I can't choose Gemini Pro 1.5 because it's not available yet, and who knows when it will be available or how good it'll be." Then when they make their next announcement, I'll be even less likely to give it any attention.

bobvanluijt2• 2 years ago

I have access and will share some learnings soon

whywhywhywhy• 2 years ago

After the complete farce that was the last 90% faked video of their tech, maybe just give us a text box we can talk to the thing and see it working ourselves next time.

Like it's shocking to me, are management really so clueless they don't realize how far behind they are? This isn't 2010 Google, your not the company that made your success anymore and in a decade the only two sure fire things that will still exist are android and chrome. Search, Maps, Youtube are all in precarious positions that the right team could dethrone.

summerlight3• 2 years ago

I believe this is a standard practice in Google whenever they need to launch a change expected to consume huge resources and they cannot reasonably predict the demand. Though I agree that this is a bad PR practice; waitlist should be considered as a compromise, not a PR technique.

crazygringo4• 2 years ago

These announcements are mainly for investors and other people interested in planning purposes. It's important to know the roadmap. More information is better.

I get that it's frustrating not to be able to play with it immediately, but that's just life. Announcing things in advance is still a valuable service for a lot of people.

Plus tons of people have been claiming that Google has somehow fallen behind in the AI race, so it's important for them to counteract that narrative. Making their roadmap more visible is a legitimate strategy for that.

dpkirchner2• 2 years ago

I wrote off the PS5 because of waitlists. I was surprised to learn just yesterday that they are now actually, honestly purchasable (what I would consider "released").

I guess I let my original impression anchor my long-term feelings about the product. Oh well.

TheFragenTaken• 2 years ago

It's probably going to be dead/deprecated in a year, so maybe there's a silver lining to how hard it is to get to use the service. I, for one, wouldn't "build with Gemini".

animex• 2 years ago

I don't think I've ever engaged with a product after "joining their waitlist". By the time they end up utilizing that funnel, competitors have already released feature upgrades or new products cannibalizing their offering.

alphabetting• 2 years ago

Massive whoa if true from technical report

"Studying the limits of Gemini 1.5 Pro's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens"

https://storage.googleapis.com/deepmind-media/gemini/gemini_...

stavros2• 2 years ago

Until I can talk to it, I care exactly zero.

peterisza• 2 years ago

you can buy their stock if you think they'll make a lot of money with their tech

megaman8212• 2 years ago

So, will this outperform any RAG approach as long as the data fits inside the context window?

ArcaneMoose• 2 years ago

Cost would still be a big concern

saliagato• 2 years ago

basically, yes. Pinecone? Dead. Azure AI Search? Dead. Quadrant? Dead.

TheGeminon• 2 years ago

Outperform is dependent on the RAG approach (and this would be a RAG approach anyways, you can already do this with smaller context sizes). A simplistic one, probably, but dumping in data that you don't need dilutes the useful information, so I would imagine there would be at least _some_ degradation.

But there is also the downside of "tuning" the RAG to return less tokens you will miss extra context that could be useful to the model.

CuriouslyC• 2 years ago

A perfect RAG system would probably outperform everything in a larger context due to prompt dilution, but in the real world putting everything in context will win a lot of the time. The large context system will also almost certainly be more usable due to elimination of retrieval latency. The large context system might lose on price/performance though.

chasd00• 2 years ago

are you going to upload 10M tokens to Gemini on every request? That's a lot of data moving around when the user is expecting a near realtime response. Seems like it would still be better to only set the context with information relevant to the user's prompt which is what plain rag does.

Workaccount2• 2 years ago

10M tokens is absolutely jaw dropping. For reference, this is approximately thirty books of 500 pages each.

Having 99% retrieval is nuts too. Models tend to unwind pretty badly as the context (tokens) grows.

Put these together and you are getting into the territory of dumping all your company documents, or all your departments documents into a single GPT (or whatever google will call it) and everyone working with that. Wild.

kranke155• 2 years ago

Seems like Google caught up. Demis is again showing an incredible ability to lead a team to make groundbreaking work.

matsemann• 2 years ago

Could you (or someone) explain what this means?

FergusArgyll3• 2 years ago

The input you give it can be very long. This can qualitatively change the experience. Imagine, for example, copy pasting the entire lord of the rings plus another 100 books you like and asking it to write a similar book...

ehsankia• 2 years ago

It's how much text it can consider at a time when generating a response. Basically the size of the prompt. A token is not quite a word but you can think of it as roughly that. Previously, the best most LLMs could do is around 32K. This new model does 1M, and in testing they could put it up to 10M with near perfect retrieval.

As the other comment mentions, you can paste the content of entire books or documents and ask very pointed question about it. Last year, Anthropic was showing off their 100K context window, and that's exactly what they did, they gave it the content of The Great Gatsby and asked it questions about specific lines of the book.

Similarly, imagine giving it hundreds of documents and asking it to spot some specific detail in there.

famouswaffles5• 2 years ago

Another whoa for me

Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.

Results - https://imgur.com/a/qXcVNOM

usaar3332• 2 years ago

I think this somewhat is mostly due to the ability to handle high context lengths better. Note how Claude 2.1 already highly outperforms GPT-4 on this task.

cchance9• 2 years ago

Did you watch the video of the Gemini 1.5 video recall after it processed the 44 minute video... holy shit

ranulo• 2 years ago

This new generation also delivers a breakthrough in long-context understanding. We’ve been able to significantly increase the amount of information our models can process — running up to 1 million tokens consistently, achieving the longest context window of any large-scale foundation model yet.

Sweet, this opens up so many possibilities.

tsunamifury• 2 years ago

Google is like a nervous and insecure engineer — blowing their value by rushing the narrative and releasing too much too confusingly fast.

sho_hn13• 2 years ago

When OpenAI raced through 3/3.5/4 it was "this team ships" and excitement.

This cargo-cult hate train is getting tiresome. Half the comments on anything Google-related are like this now, and it doesn't add anything to the conversation.

epiccoleman• 2 years ago

The difference, though, as someone who really doesn't have a particular dog in this fight, is that I can go use GPT-4 right now, and see for myself whether it's as exciting as the marketing materials say.

mynameisvlad2• 2 years ago

Gemini Ultra was announced two months ago. It just launched in the last week. It literally is still the featured post on the AI section of their blog, above this announcement. https://blog.google/technology/ai/

There’s “this team ships” and there’s “ok maybe wait until at least a few people have used your product before you change it all”.

moralestapia3• 2 years ago

"this team ships"

Because they actually shipped ... (!)

SushiHippie• 2 years ago

Does this mean gemini ultra 1.0 -> gemini ultra 1.5 is the same as gpt-4 -> gpt-4-turbo?

hackerlight• 2 years ago

There's no Gemini Ultra 1.5 yet. Gemini Pro 1.5 is a smaller model than Gemini Ultra 1.0.

prakhar897• 2 years ago

Can anyone explain how context length is tested? Do they prompt something like:

"Remember val="XXXX" .........10M tokens later....... Print val"

NhanH• 2 years ago

Yep, that’s actually a common one

blovescoffee• 2 years ago

Very simplified There are arrays (matrices) that are length 10M inside the model.

It’s difficult to make that array longer because training time explodes.

halflings• 2 years ago

Yep that's pretty much it! That's what they call needle in a haystack. See: https://github.com/gkamradt/LLMTest_NeedleInAHaystack

cchance9• 2 years ago

yep they hide things throughout the prompt and then ask it about that specific thing, imagine hiding passwords in a giant block of text and then being like, what was bobs password 10 million tokens later.

According to this it's remembering with 99% accuracy, which if you think about it is NUTS, can you imagine reading a 22x 1000 page books, and remembering every single word that was said with 100% accuracy lol

foota2• 2 years ago

Interestingly, there's a decent chance I'd remember if there was an out of context passage saying "the password is FooBar". I wonder if it would be better to test with minor edits? E.g., "what color shirt was X wearing when..."

phoe18• 2 years ago

The branding is very confusing, shouldn't this be Gemini Pro 1.5 since the most capable model is called Ultra 1.0?

blurbleblurble• 2 years ago

Extremely confusing!

butler14• 2 years ago

Maybe they use their own generative AI to do their branding

dkjaudyeqooe3• 2 years ago

Can anyone lay out the various models and their features or point to a resource?

I asked the free model (whatever that is) and it wasn't very helpful, alterating betweens a sales bot for Ultra and being somewhat confused itself.

Edit: apparently it goes 1.0 Pro, 1.0 Ultra, 1.5 Pro, 1.5 Ultra and so on.

Alifatisk12• 2 years ago

Here's the models, https://news.ycombinator.com/item?id=39304270 This is about Gemini Pro going from version 1.0 to 1.5, nothing else.

Gemini ultra 1.0 is still on version 1.0

UncleMeat• 2 years ago

Google is somehow truly awful at this. I thought it was funny when branding messes happened in 2017. I cried when they announced "Google Meet (original)." Now I don't even know what to do.

I'm stunned that Google hasn't appointed some "name veto person" that can just say "no, you aren't allowed to have three different things called 'Gemini Advanced', 'Gemini Pro', and 'Gemini Ultra.'" Like surely it just takes Sundar saying "this is the stupidest fucking thing I've ever seen" to some SVP to fix this.

meowface3• 2 years ago

And somehow the more advanced one is still on 1.0 (for now) and the less advanced one is on 1.5.

seydor5• 2 years ago

We will ask what its real name is as soon as it becomes sentient

iamdelirium2• 2 years ago

No? Do you call it the iPhone Pro 15 or the iPhone 15 Pro? Their naming makes sense if you follow most consumer technology.

summerlight3• 2 years ago

This is something close to CPU versioning. You have two axis; performance branding and its generation. Nano, Pro and Ultra is something similar to i3, i5 and i7. The numbered versions 1.0, 1.5, ... can be mapped to 13th gen, 14th gen, ... so on. And people usually don't need to understand the generation part this unless they're enthusiasts.

arange• 2 years ago

signup on mobile too big, doesn't fit submit button :\

guybedo• 2 years ago

looks interesting enough that i wanted to give Gemini a try and join the waitlist.

And i thought it would be easy, what a rookie mistake.

Looks like "France" isn't on the list of available regions for Ai Studio ?

Now i'm trying to use Vertex AI, not even sure what's the difference with Ai Studio, but it seems it's available.

So far i've been struggling for 15 minutes through a maze of google cloud pages: console, docs, signups. No end in sight, looks like i won't be able to try it out

IanCal2• 2 years ago

It's not available outside of a private preview yet. The page says you can use 1.0 ultra in vertex but it's not available to me in the UK.

I can't get on the waitlist, because the waitlist link redirects to aistudio and I can't use that.

I should stop expecting that I can use literally anything google announces.

simonw• 2 years ago

I'd love to know how much a 1 million token prompt is likely to cost - both in terms of cash and in terms of raw energy usage.

bearjaws2• 2 years ago

Cannot emphasize enough, even with the improvements in context handling I imagine 128k tokens costs as much as 16k tokens did previously.

So 1M tokens is going to be astronomical.

empath-nirvana5• 2 years ago

When you account for this, you have to consider how much it would cost to have a human perform the same task.

foliveira• 2 years ago

"Gemini 1.5 Pro (...) matches or surpasses Gemini 1.0 Ultra’s state-of-the-art performance across a broad set of benchmarks."

So Pro is better than Ultra, but only if the version numbers are higher?

denysvitali• 2 years ago

Yes, but you'd have to wait for Gemini Pro Max next year to see the real improvements

renewiltord2• 2 years ago

Isn't that usually the case with many products? Like the M3 Pro CPU in the new Macs is more powerful than the M1 Max in the old Macs.

The Nano < Pro < Ultra is an in-revision thing. For their LLMs it's a size thing. Then there's newer releases of Nano, Pro, and Ultra. Some Pro might be better than some older Ultra.

A lot of people seem confused about this but it feels so easy to understand that it's confusing to me that anyone could have trouble.

devindotcom• 2 years ago

Apple didn't release the M3 Pro a week after the M1 Max

thiago_fm• 2 years ago

I like that they are rushing with this and don't care enough to make it Gemini 2 or even really release it, to me it looks like they are concerned to share progress.

Hope they do a good job and once OpenAI releases GPT 5 they are competitive with it with their offerings, it will be better for everyone.

kaspermarstal• 2 years ago

Incredible. RAG will be obsolete in a year or two.

hackernoteng• 2 years ago

It's already obsolete. It doesn't work except for trivial cases which have no real value.

jeanloolz• 2 years ago

Obsolete if you don't take cost in consideration. Having 10 millions of token going through each layer of the LLM is going to cost a lot of money each time. At gpt4 rate that could mean 200 dollars for each inference

scarmig3• 2 years ago

The technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_...

jpeter3• 2 years ago

OpenAI has no Moat

jklinger410• 2 years ago

They only have a head start, and the lead is closing

seydor5• 2 years ago

hence why it's Open

rvz2• 2 years ago

This. He’s right you know.

OpenAI is extremely overvalued and Google is closing their lead rapidly.

fnordpiglet• 2 years ago

Is there any meaningful valuation on OpenAI? It’s not for sale, there is no market.

Google … has no ability to commercialize anything. Their only commercial successes are ads and YouTube. Doing deceptive launches and flailing around with Gemini isn’t helping their product prospects. I wouldn’t take a bet between open ai and anyone, but I also wouldn’t take a bet on Google succeeding commercially on anything other than pervasive surveillance and adware.

wrsh07• 2 years ago

A reference to the good doc: https://www.semianalysis.com/p/google-we-have-no-moat-and-ne...

While I'm linking semianalysis, though, it's probably worth talking about how everyone except Google is GPU poor: https://www.semianalysis.com/p/google-gemini-eats-the-world-... (paid)

Whether Google has the stomach to put these models out publicly without neutering their creativity or their existing business model is a different discussion.

Google has a serious GPU (well, TPU) build out, and the fact that they're able to train moe models on it means there aren't any technical barriers preventing them from competing at the highest levels

Keyframe5• 2 years ago

they also have internet.zip and all of its repo history as well as usenet and mails etc.. which others don't.

anonyfox• 2 years ago

but GPT-4 is nearly a year old now, I'd wait for the next release of OAI before judgement. Probably rather soonish now I would expect.

jpeter3• 2 years ago

You were right i guess :)

gpjanik• 2 years ago

0 trust to what they put out until I see it live. After the last "launch" video which was fundamentally a marketing edit not showing the real product, I don't trust anything coming out of Google that isn't an instantly testable input form.

tfsh• 2 years ago

The videos shown in these demos have clearly learnt from that as they're using a real live product, filmed on their computers with timers in the bottom showing how long the computations take.

frays• 2 years ago

I completely share the same views as you after their last video - and it appears that they've learnt their lesson this time.

If you watch the videos in the blog post, you can see it's a screen recording on a computer without any editing/stitching of different scenes together.

It's good to be sceptical but as engineers we should all remain open.

replwoacause2• 2 years ago

100%. Google continues to underwhelm. Not buying it until I can try it.

dingclancy• 2 years ago

Essentially, the focus seems to be on leveraging the media buzz around Gemini 1.0 by highlighting the development of version 1.5. While GPT-4's position relative to Gemini 1.5 remains unclear, and the specifics of ChatGPT 4.5 are yet to be disclosed, it's worth noting that no official release has taken place until the functionality is directly accessible in user chats.

Google appears to be making strides in catching up.

When it comes to my personal workflow and accomplishing tasks, I still find ChatGPT to be the most effective tool. My familiarity with its features has made it indispensable. The integration of mentions and tailored GPTs seamlessly enhances my workflow.

While Gemini may match the foundational capabilities of LLMs, it falls short in delivering a product that efficiently aids in task completion.

vacuumcl2• 2 years ago

I don't mean this in a bad way, but when I read a comment like yours which includes phrases like "seamlessly enhances my workflow" and "efficiently aids in task completion", I can't help but feel like it's ChatGPT-generated, and if so I think it's a shame, just write like yourself.

But maybe you do, and I am seeing patterns in sand.

Breza4• 2 years ago

Google appears to be making strides in catching up.

I say it's even more than that. OpenAI had a bigger lead when it released GPT-2 than it does now. They're burning through cash to try to hold on to a lead of a few months over the competition.

losvedir• 2 years ago

If I understand correctly, they're releasing this for Pro but not Ultra, which I think is akin to GPT 3.5 vs 4? Sigh, the naming is confusing...

But my main takeaway is the huge context window! Up to a million, with more than 100k tokens right now? Even just GPT 3.5 level prediction with such a huge context window opens up a lot of interesting capabilities. RAG can be super powerful with that much to work with.

danpalmer2• 2 years ago

The announcement suggests that 1.5 Pro is similar to 1.0 Ultra.

benopal64• 2 years ago

I am reaching a bit, however, I think its a bit of a marketing technique. The Pro 1.5 being compared to the Ultra 1.0 model seems to imply that they will be releasing a Ultra 1.5 model which will presumably have similar characteristics to the new Pro 1.5 model (MOE architecture w/ a huge context window).

ygouzerh• 2 years ago

So Pro and Ultra are from my understanding link to the number of parameters. More parameters means more reasonning capabilities, but more compute needed.

So Pro is like the light and fast version and Ultra the advanced and expensive one.

cchance9• 2 years ago

It's sizes

Nano/Pro/Ultra are model SIZES. 1.0/1.5 is generations of the architecture.

amf122• 2 years ago

Maybe this analogy would help: iPhone 15, iPhone Pro 15, iPhone Pro Max 15 and then iPhone Pro 15.5

golergka• 2 years ago

In one of the demos, it successfully navigates a threejs demo and finds the place to change in response to a request.

How long until it shows similar results on middle-sized and large codebases? And do the job adequately?

kypro• 2 years ago

1-2 years probably. There will still be a question around who determines what "adequately" is for a while though. Presumably even if an LLM can do something in theory you wouldn't actually want it doing anything without human oversight.

And we should keep in mind that to understand a code change in depth is often just as much work as making the change. When review PRs I don't really know exactly what every change is doing. I certain haven't tested it to be 100% certain I understand fully. I'm just checking the logic looks mostly right and that I don't see anything clearly wrong, and even then I'll often need to ask for clarifications why something was done.

I can't imagine LLMs being used in most large code bases for a while yet. They'd probably need to be 99.9% reliable before we can start trusting them to make changes without verifying every line.

simon_kun• 2 years ago

Today.

pryelluw• 2 years ago

Gemini (or whatever google ai) will be all about ads. I’m not adopting this shit. Their whole business model is ads. Why would I adopt a product from a company that only cares about selling more ads?

Alifatisk12• 2 years ago

Google One's business model is not ads?

I mention Google One because you can access Gemini Ultra through it.

imp0cat2• 2 years ago

All their services are just a way to get more information about their users so they can serve them ads.

Those Gemini queries will be no exception.

snapcaster• 2 years ago

Agreed, people continually forget that Google has fundamentally failed at everything besides selling ads despite decades of moonshots and other attempts to shift the business. Very skeptical that any company getting 80% revenue from ads will be able to resist the pressure to advertise

royletron• 2 years ago

Is there a reason this isn't available in the UK/France/Germany/Spain but is in available in Jersey... and Tuvalu?

onlyrealcuzzo• 2 years ago

EU regulations and fines.

vibrolax• 2 years ago

Probably because EU/national governments have regulations with respect to the safety and privacy of the users, and the purveyors must evaluate the performance of their products against the regulatory standards.

seydor5• 2 years ago

Onwards to a billion tokens

fernandotakai• 2 years ago

i saw this announcement on twitter and i was excited to check it out, only to see that "we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI".

please google, only announce things when people can actually use it.

xyzzy_plugh2• 2 years ago

I miss when I didn't have to scroll to read a single tweet.

ComputerGuru2• 2 years ago

Twitter has that functionality natively now, but I don't know if you have to be a pro user to access. It's the book icon in the upper-right corner of the first tweet in a series. Links to this, but it looks different when I view it in incognito vs logged in: https://twitter.com/JeffDean/thread/1758146022726041615

xyzzy_plugh2• 2 years ago

The functionality I'm talking about is tweets not being walls of text that require scrolling to read. I have no idea what you're describing.

famouswaffles5• 2 years ago

Finally, we highlight surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person learning from the same content.

Results - https://imgur.com/a/qXcVNOM

From the technical report https://storage.googleapis.com/deepmind-media/gemini/gemini_...

poulpy123• 2 years ago

at a similar level to a person learning from the same content.

That's an incredibly low bar

ithkuil2• 2 years ago

It's incredible how fast goalposts are moving.

The same feat one year ago would have been almost unbelievable.

KeplerBoy2• 2 years ago

Since when are we expecting super-human capabilities?

elevatedastalt3• 2 years ago

:muffled sounds of goalposts being shifted in the distance:

Just a few years ago we used to clap if an NLP model could handle negation reliably or could generate even a paragraph of text in English that was natural sounding.

Now we are at a stage where it is basically producing reams of natural sounding text, performing surprisingly well on reasoning problems and translation of languages with barely any data despite being a markov chain on steroids, and what does it hear? "That's an incredibly low bar".

zacmps• 2 years ago

The author (the human learner) has some formal experience in linguistics and has studied a variety of languages both formally and informally, though no Austronesian or Papuan languages

From the language benchmark (parentheses mine).

JyB• 2 years ago

Jarring you're not adding more context to your comment.

newzisforsukas3• 2 years ago

you are insane if you actually think this.

lukemelas2• 2 years ago

Author here (of the Kalamang paper). One of my coauthors was the human baseline and he spent many months reading the grammar book (and is incredibly talented at learning languages). It’s really a very high bar.

seydor5• 2 years ago

what if we ask it to translate an undeciphered language

dougmwne2• 2 years ago

It produces basically random translations. This is covered in the 0-shot case where no translation manual was included in the context. Due to how rare this language is, it’s essentially untranslated in the training corpus.

famouswaffles5• 2 years ago

If you mean to dump random passages of text with no parallel corpora or grammar instructions then it won't do better than random.

That said, I think that if you gave a LLM language text to predict during training, I believe that even if no parallel corpora exists during training, we could have a LLM that could still translate that language to some other language it also trained on.

lukemelas2• 2 years ago

Author of the Kalamang paper here. We’ve thought about this a good amount (e.g. there are interesting Mesoamerican scripts), but ultimately we decided to work on low-resource languages, as they’re much more useful. It’s also possible to use them as an evaluation benchmark, which isn’t really possible if nobody speaks the language. We’d like to expand the scope beyond Kalamang, though, and maybe at some point we’ll investigate an undeciphered language as well.

uptownfunk• 2 years ago

Google is a public company. Anything and everything will be scrutinized very heavily by shareholders. Of course how Zuck operates very different than Sindar.

What are they doing with their free cash is my question. Are they waiting for the LLM bubble to pop to buy some of these companies at a discount?

ComputerGuru2• 2 years ago

The context window size - if it really works as advertised - is pretty ground-breaking. It would replace the need to RAG or fine tune for one-off (or few-off) analys{is,es} of input streams cheaper and faster. I wonder how they got past the input token stuffing problems everyone else runs into.

jcuenod• 2 years ago

It won't remove the use of RAG at all. That's like saying, "wow, now that I've upgraded my 128GB HDD to 1TB, I'll never run out of space again."

madisonmay• 2 years ago

It's more like saying "I've upgraded to 128GB of RAM, I'll never use my disk again".

sebzim45003• 2 years ago

10 TB for an accurate proportion.

And I think people who buy a laptop with a 1TB SSD generally don't run out of space, at least I don't.

lumost• 2 years ago

They are almost certainly using some form of sparse attention. If you linearize the attention operation, you can scale up to around 1-10M tokens depending on hardware before hitting memory constraints. Linearization works off the assumption that for a subsequence of X tokens out M tokens, where M os much greater than X there are likely only K tokens which are useful for the attention operation.

There are a bunch of techniques to do this, but it's unclear how well any of them scale.

ein0p2• 2 years ago

Not "almost", but certainly. Dense attention is quadratic, not even Google would be able to run it at an acceptable speed. Their model is not recurrent - they did not have the time yet (or resources - believe it or not, Google of 2023-24 is very compute constrained) to train newer SSM or recurrent based models at practical parameter counts. Then there's the fact that those models are far harder to train due to instabilities, which is one of the reasons why you don't yet see FOSS recurrent/SSM models that are SOTA at their size or tokens/sec. With sparse attention, however, long context recall will be far from perfect, and the longer the context the worse the recall. That's better than no recall at all (as in a fully dense attention model which will simply lop off the preceding parts of the conversation), but not by a hell of a lot.

popinman322• 2 years ago

vs RAG: RAG is good for searching across >billions of tokens and providing up-to-date information to a static model. Even with huge context lengths it's a good idea to submit high quality inputs to prevent the model from going off on tangents, getting stuck on contradictory information, etc..

vs fine tuning: smaller, fine-tuned models can perform better than huge models in a decent number of tasks. Not strictly fine-tuning, but for throughput limited tasks it'll likely still be better to prune a 70B model down to 2B, keeping only the components you need for accurate inference.

I can see this model being good for taking huge inputs and compressing them down for smaller models to use.

nbardy• 2 years ago

RAG will stick around, at some point you want to retrieve grounded information samples to inject in the context window. RAG+long context just gives you more room for grounded context.

Think building huge relevant context on topics before answering.

torginus2• 2 years ago

Tbh, I haven't read the paper, but I think it's pretty self-evident that large contexts aren't cheap - the AI has to comb through every word of the context for each successive generated token at least once, so it's going to be at least linear.

Havoc2• 2 years ago

Saw testing earlier that suggested the context does indeed work right

Alifatisk12• 2 years ago

I remember one of the biggest advantages with Google Bard was the heavily limited context window. I am glad Google is now actually delivering some exciting news now with Gemini and this gigantic token size.

Sure it's a bummer that they slap the "Join the waiting list", but it's still interesting to read about their progress and competing with ClosedAi (OpenAi).

One last thing I hope they fix is the heavily morally and ethically guardrail, sometimes I can barely ask proper questions without it triggering Gemini to educate me about what's right and wrong. And when I try the same prompt with ChatGPT and Bing ai, they happily answer.

elevatedastalt3• 2 years ago

"biggest advantages with Google Bard"

Did you mean disadvantages?

Alifatisk12• 2 years ago

Yes, thanks.

CrypticShift• 2 years ago

Most data accumulates gradually (e.g., one email at a time, one line of text at a time across various documents). Is this huge 10M scale of context window relevant to a gradual, yet constant, influx of data (like a prompt over a whole google workspace) ?

Imnimo• 2 years ago

This is the first time I've been legitimately impressed by one of Google's LLMs (with the obvious caveat that I'm taking the results reported in their tech report at face value).

replwoacause2• 2 years ago

It’s just marketing at this point, nothing to be impressed by. It’s a mistake to take at face value.

sremani• 2 years ago

I have Gemini Advanced, do I get access to this? Google is giving Microsoft run for money for branding confusion.

Alifatisk12• 2 years ago

Not yet, Gemini advanced is using Gemini Ultra, not Gemini pro.

Ecstatify3• 2 years ago

Gemini advanced is terrible.

I asked it to rephrase "Are the original stated objectives still relevant?"

It's starts going on about Ukraine and Russia.

https://g.co/gemini/share/ddb3887f79e2

piva00• 2 years ago

I thought I wouldn't but I'm getting really, really confused with the naming and branding of what Gemini is a model and which is a product. Advanced, Pro, Ultra, seemingly Pro is getting better than Ultra? And Advanced is the product using the Ultra underlying model?

Ugh, my brain.

tapoxi• 2 years ago

I've read this sentence three times, wow what horrible branding.

vessenes• 2 years ago

The white paper is worth a read. The things that stand out to me are:

1. They don't talk about how they get to 10M token context

2. They don't talk about how they get to 10M token context

3. The 10M context ability wipes out most RAG stack complexity immediately. (I imagine creating caching abilities is going to be important for a lot of long token chatting features now, though). This is going to make things much, much simpler for a lot of use cases.

4. They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting.

5. It seems like 1.5 Ultra is going to be highly capable. 1.5 Pro is already very very capable. They are running up against very high scores on many tests, and took a minute to call out some tests where they scored badly as mostly returning false negatives.

Upshot, 1.5 Pro looks like it should set the bar for a bunch of workflow tasks, if we can ever get our hands on it. I've found 1.0 Ultra to be very capable, if a bit slow. Open models downstream should see a significant uptick in quality using it, which is great.

Time to dust out my coding test again, I think, which is: "here is a tarball of a repository. Write a new module that does X".

I really want to know how they're getting to 10M context, though. There are some intriguing clues in their results that this isn't just a single ultra-long vector; for instance, their audio and video "needle" tests, which just include inserting an image that says "the magic word is: xxx", or an audio clip that says the same thing, have perfect recall across up to 10M tokens. The text insertion occasionally fails. I'd speculate that this means there is some sort of compression going on; a full video frame with text on it is going to use a lot more tokens than the text needle.

CharlieDigital3• 2 years ago

    > The 10M context ability wipes out most RAG stack complexity immediately.

Remains to be seen.

Large contexts are not always better. For starters, it takes longer to process. But secondly, even with RAG and the large context of GPT4 Turbo, providing it a more relevant and accurate context always yields better output.

What you get with RAG is faster response times and more accurate answers by pre-filtering out the noise.

behnamoh2• 2 years ago

Don't forget that Gemini also has access to the internet, so a lot of RAGging becomes pointless anyway.

killerstorm• 2 years ago

Hopefully we can get a better RAG out of it. Currently people do incredibly primitive stuff like chunking text into chunks of a fixed size and adding them to vector DB.

An actually useful RAG would be to convert text to Q&A and use Q's embeddings as an index. Large context can make use of in-context learning to make better Q&A.

cs7022• 2 years ago

1. They don't talk about how they get to 10M token context
2. They don't talk about how they get to 10M token context

Yes. I wonder if they're using a "linear RNN" type of model like Linear Attention, Mamba, RWKV, etc.

Like Transformers with standard attention, these models train efficiently in parallel, but their compute is O(N) instead of O(N²), so in theory they can be extended to much longer sequences much efficiently. They have shown a lot of promise recently at smaller model sizes.

Does anyone here have any insight or knowledge about the internals of Gemini 1.5?

candiodari2• 2 years ago

They do give a hint:

"This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture."

One thing you could do with MoE is giving each expert different subsets of the input tokens. And that would definitely do what they claim here: it would allow search. You want to find where someone said "the password is X" in a 50 hour audio file, this would be perfect.

If your question is "what is the first AND last thing person X said" ... it's going to suck badly. Anything that requires taking 2 things into account that aren't right next to eachother is just not going to work.

sebzim45003• 2 years ago

The fact they are getting perfect recall with millions of tokens rules out any of the existing linear attention methods.

usaar3332• 2 years ago

They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting.

They try to push that, but it's not the most convincing. Look at Table 8 for text evaluations (math, etc.) - they don't even attempt a comparison with GPT-4.

GPT-4 is higher than any Gemini model on both MMLU and GSM8K. Gemini Pro seems slightly better than GPT-4 original in Human Eval (67->71). Gemini Pro does crush naive GPT-4 on math (though not with code interpreter and this is the original model).

All in 1.5 Pro seems maybe a bit better than 1.0 Ultra. Given that in the wild people seem to find GPT-4 better for say coding than Gemini Ultra, my current update is Pro 1.5 is about equal to GPT-4.

But we'll see once released.

cchance9• 2 years ago

I mean i don't see GPT4 watching a 44 minute movie and being able to exactly pinpoint a guy taking a paper out of his pocket..

panarky3• 2 years ago

people seem to find GPT-4 better for say coding than Gemini Ultra

For my use cases, Gemini Ultra performs significantly better than GPT-4.

My prompts are long and complex, with a paragraph or two about the general objective followed by 15 to 20 numbered requirements. Often I'll include existing functions the new code needs to work with, or functions that must be refactored to handle the new requirements.

I took 20 prompts that I'd run with GPT-4 and fed them to Gemini Ultra. Gemini gave a clearly better result in 16 out of 20 cases.

Where GPT-4 might miss one or two requirements, Gemini usually got them all. Where GPT-4 might require multiple chat turns to point out its errors and omissions and tell it to fix them, Gemini often returned the result I wanted in one shot. Where GPT-4 hallucinated a method that doesn't exist, or had been deprecated years ago, Gemini used correct methods. Where GPT-4 called methods of third-party packages it assumed were installed, Gemini either used native code or explicitly called out the dependency.

For the 4 out of 20 prompts where Gemini did worse, one was a weird rejection where I'd included an image in the prompt and Gemini refused to work with it because it had unrecognizable human forms in the distance. Another was a simple bash script to split a text file, and it came up with a technically correct but complex one-liner, while GPT-4 just used split with simple options to get the same result.

For now I subscribe to both. But I'm using Gemini for almost all coding work, only checking in with GPT-4 when Gemini stumbles, which isn't often. If I continue to get solid results I'll drop the GPT-4 subscription.

spott2• 2 years ago

Gemini Pro seems slightly better than GPT-4 original in Human Eval (67->71).

Though they talk a bunch about how hard it was to filter out Human Eval, so this probably doesn't matter much.

swalsh• 2 years ago

"The 10M context ability wipes out most RAG stack complexity immediately."

I'm skeptical, my past experience is just becaues the context has room to stuff whatever you want in it, the more you stuff in the context the less accurate your results are. There seems to be this balance of providing enough that you'll get high quality answers, but not too much that the model is overwhelmed.

I think a large part of developing better models is not just a better architectures that support larger and larger context sizes, but also capable models that can properly leverage that context. That's the test for me.

HereBePandas• 2 years ago

They explicitly address this in page 11 of the report. Basically perfect recall for up to 1M tokens; way better than GPT-4.

swyx• 2 years ago

also costs are always based on context token, you dont want to put in 10m of context for every request (its just nice to have that option when you want to do big things that dont scale)

chuckcode• 2 years ago

Would like to see the latency and cost of parsing entire 10M context before throwing out the RAG stack which is relatively cheap and fast.

tkellogg• 2 years ago

costs rise on a per-token basis. So you CAN use 10M tokens, but it's probably not usually a good idea. A database lookup is still better than a few billion math operations.

theolivenbaum• 2 years ago

Also unless they significantly change their pricing model, we're talking about 0.5$ per API call at current prices

aik• 2 years ago

Have to consider cost for all of this. Big value of RAG already even given the size of GPT-4’a largest context size is it decreases cost very significantly.

patja• 2 years ago

I think there are also a lot of people who are only interested in RAG if they can self-host and keep their documents private.

koliber2• 2 years ago

LLMs are able to utilize “all the worlds” knowledge during training and give seemingly magical answers. While providing context in the query is different than training models, is it possible that more context will give more materials to the LLM and it will be able to pick out the relevant bits on its own?

What if it was possible, with each query, to fine tune the model on the provided context, and then use that JIT fine-tuned model to answer the query?

freedomben2• 2 years ago

Is 10M token context correct? The blog post I see 1M but I'm not sure if these are different things

Edit: Ah, I see, it's 1M reliably in production, up to 10M in research:

Through a series of machine learning innovations, we’ve increased 1.5 Pro’s context window capacity far beyond the original 32,000 tokens for Gemini 1.0. We can now run up to 1 million tokens in production.
This means 1.5 Pro can process vast amounts of information in one go — including 1 hour of video, 11 hours of audio, codebases with over 30,000 lines of code or over 700,000 words. In our research, we’ve also successfully tested up to 10 million tokens.

huytersd4• 2 years ago

I know how I’m going to evaluate this model. Upload my codebase and ask it to “find all the bugs”.

p1esk• 2 years ago

How could one hour of video fit in 1M tokens? 1 hour at 30fps is 3600*30=100k frames. Each frame is converted in 256 tokens. So either they are not processing each frame, or each frame is converted into fewer tokens.

tbruckner• 2 years ago

How do you know it isn't RAG?

tveita• 2 years ago

The 10M context ability wipes out most RAG stack complexity immediately.

The video queries they show take around 1 minute each, this probably burns a ton of GPU. I appreciate how clearly they highlight that the video is sped up though, they're clearly trying to avoid repeating the "fake demo" fiasco from the original Gemini videos.

theGnuMe• 2 years ago

For #1 and #2 it is some version of mixture of experts. This is mentioned in the blog post. So each expert only sees a subset of the tokens.

I imagine they have some new way to route tokens to the experts that probably computes a global context. One scalable way to compute a global context is by a state space model. This would act as a controller and route the input tokens to the MoEs. This can be computed by convolution if you make some simplifying assumptions. They may also still use transformers as well.

I could be wrong but there are some Mamba-MoEs papers that explore this idea.

resouer• 2 years ago

The 10M context ability wipes out most RAG stack complexity immediately.

This may not be true. My experience of the complexity of RAG lays in how to properly connect to various unstructured data sources and perform data transformation pipeline for large scale data set (which means GB, TB or even PB). It's in the critical path rather a "nice to have", because the quality of data and the pipeline is a major factor for the final generated the result. i.e., in RAG, the importance of R >>> G.

jorvi• 2 years ago

I just hope at some point we get access to mostly uncensored models. Both GPT-4 and Gemini are extremely shackled, and a slightly inferior model that hasn’t been hobbled by a very restricting preprompt would handily outperform them.

ShamelessC• 2 years ago

You can customize the system prompt with ChatGPT or via the completions API, just fyi.

ren_engineer• 2 years ago

RAG would still be useful for cost savings assuming they charge per token, plus I'm guessing using the full-context length would be slower than using RAG to get what you need for a smaller prompt

nostrebored• 2 years ago

This is going to be the real differentiator.

HN is very focused on technical feasibility (which remains to be seen!), but in every LLM opportunity, the CIO/CFO/CEO are going to be concerned with the cost modeling.

The way that LLMs are billed now, if you can densely pack the context with relevant information, you will come out ahead commercially. I don't see this changing with the way that LLM inference works.

Maybe this changes with managed vector search offerings that are opaque to the user. The context goes to a preprocessing layer, an efficient cache understands which parts haven't been embedded (new bloom filter use case?), embeds the other chunks, and extracts the intent of the prompt.

cchance9• 2 years ago

The youtube video of the Multimodal analysis of a video is insane, imagine feeding in movies or tv shows and being able to autosummary or find information about them dynamically, how the hell is all this possible already? AI is moving insanely fast.

vineyardmike5• 2 years ago

imagine feeding in movies or tv shows

Google themselves have such a huge footprint of various businesses, that they alone would be an amazing customer for this, never mind all the other cool opportunities from third parties...

Imagine that they can ingest the entirety of YouTube and then dump that into Google Search's index AND use it to generate training data for their next LLM.

Imagine that they can hook it up to your security cameras (Nest Cam), and then ask questions about what happened last night.

Imagine that you can ask Gemini how to do something (eg. fix appliance), and it can go and look up a YouTube video on how to accomplish that ask, and explain it to you.

Imagine that it can apply summarization and descriptions to every photo AND video in your personal Google Photos library. You can ask it to find a video of your son's first steps, or a graduation/diploma walk for your 3rd child (by name) and it can actually do that.

Imagine that Google Meet video calls can have the entire convo itself fed into an LLM (live?), instead of just a transcription. You can have an AI assistant there with you that can interject and discuss, based on both the audio and video feed.

zitterbewegung• 2 years ago

RAG doesn’t go away at 10 Million tokens if you do esoteric sources like shodan API queries.

kylerush• 2 years ago

I assume using this large of a context window instead of RAG would mean the consumption of many orders of magnitude more GPU.

karmasimida2• 2 years ago

Even 1m tokens eliminate the need for RAG, unless it is for cost.

sroussey• 2 years ago

Or accuracy

77341283• 2 years ago

1 million might sound like a lot, but it's only a few megabytes. I would want RAG, somehow, to be able to process gigabytes or terabytes of material in a streaming fashion.

localhost3• 2 years ago

RE: RAG - they haven't released pricing, but if input tokens are priced at GPT-4 levels - $0.01/1K then sending 10M tokens will cost you $100.

s-macke• 2 years ago

If you think the current APIs will stay that way, then you're right. But when they start offering dedicated chat instances or caching options, you could be back in the penny region.

You probably need a couple GB to cache a conversation. That's not so easy at the moment because you have to transfer that data to and from the GPUs and store the data somewhere.

campers• 2 years ago

In the announcements today they also halved the pricing of Gemini 1.0 Pro to $0.000125 / 1K characters, which is a quarter of GPT3.5 Turbo so it could potentially be a bit lower than GPT-4 pricing.

TweedBeetle• 2 years ago

Regarding how they’re getting to 10M context, I think it’s possible they are using the new SAMBA architecture.

Here’s the paper: https://arxiv.org/abs/2312.00752

And here’s a great podcast episode on it: https://www.cognitiverevolution.ai/emergency-pod-mamba-memor...

LightMachine• 2 years ago

As a Brazilian, I approve that choice. Vambora amigos!

renonce2• 2 years ago

They don't talk about how they get to 10M token context

I don't know how either but maybe https://news.ycombinator.com/item?id=39367141

Anyway I mean, there is plenty of public research on this so it's probably just a matter of time for everyone else to catch up

albertzeyer• 2 years ago

Why do you think this specific variant (RingAttention)? There are so many different variants for this.

As far as I know, the problem in most cases is that while the context length might be high in theory, the actual ability to use it is still limited. E.g. recurrent networks even have infinite context, but they actually only use 10-20 frames as context (longer only in very specific settings; or maybe if you scale them up).

AaronFriel3• 2 years ago

There will always be more data that could be relevant than fits in a context window, and especially for multi-turn conversations, huge contexts incur huge costs.

GPT-4 Turbo, using its full 128k context, costs around $1.28 per API call.

At that pricing, 1m tokens is $10, and 10m tokens is an eye-watering $100 per API call.

Of course prices will go down, but the price advantage of working with less will remain.

77341283• 2 years ago

Would the price really increase linearly? Isn't the demands on compute and memory increasing steeper than that as a function of context length?

elorant• 2 years ago

I don't see a problem with this pricing. At 1m tokens you can upload the whole proceedings of a trial and ask it to draw an analysis. Paying $10 for that sounds like a steal.

qwerty_clicks• 2 years ago

FYI, MM is the standard for million. 10MM not 10M I’m reading all these comments confused as heck why you are excited about 10M tokens

MichaelNolan• 2 years ago

Maybe for accountants, but for everyone else a single M is much more common.

a_vanderbilt• 2 years ago

After their giant fib with the Gemini video a few weeks back I'm not believing anything til I see it used by actual people. I hope it's that much better than GPT-4, but I'm not holding my breath there isn't an asterisk or trick hiding somewhere.

nborwankar• 2 years ago

Re RAG aren’t you ignoring the fact that no one wants to put confidential company data into such LLM’s. Private RAG infrastructure remains a need for the same reason that privacy of data of all sorts remains a need. Huge context solves the problem for large open source context material but that’s only part of the picture.

outside1234• 2 years ago

It takes 60 seconds to process all of that context in their three.js demo, which is, I will say, not super interactive. So there is still room for RAG and other faster alternatives to narrow the context.

aubanel2• 2 years ago

They are pretty clear that 1.5 Pro is better than GPT-4 in general, and therefore we have a new LLM-as-judge leader, which is pretty interesting

I fully disagree, they compare Gemini 1.5 Pro and GPT4 only on context length, not on other tasks where they compare it only to other Gemini which is a strange self-own.

I'm convinced that if they do not show the results against GPT4/Claude, it is because they do not look good.

kristjansson• 2 years ago

For other's reference, the paper: https://storage.googleapis.com/deepmind-media/gemini/gemini_...

joshsabol46• 2 years ago

The 10M context ability wipes out most RAG stack complexity immediately.

RAG is needed for the same reason you don't `SELECT *` all of your queries.

nestorD2• 2 years ago

Regarding the 10M tokens context, RingAttention has been shown [0] recently (by researchers, not ML engineers in a FAANG) to be able to scale to comparable (1M) context sizes (it does take work and a lot of GPUs).

[0] https://news.ycombinator.com/item?id=39367141

jebarker• 2 years ago

researchers, not ML engineers in a FAANG

Why did you point out this distinction?

bschne• 2 years ago

The 10M context ability wipes out most RAG stack complexity immediately.

1. People mention accuracy issues with longer contexts 2. People mention processing time issues with longer contexts 3. Something people haven't mentioned in this thread is cost -- even thought prompt tokens are usually cheaper than generated tokens, and Gemini seems to be cheaper than GPT-4, putting a whole knowledge base or 80-page document in the context is going to make every time you run that prompt quite expensive

lqcfcjx• 2 years ago

This might be a stupid question - even if there's no quality degradation from 10M context, will it be extremely slow in reference?

Havoc2• 2 years ago

3. The 10M context ability wipes out most RAG stack complexity immediately.

I'd imagine RAG would still be much more efficient computationally

oblio• 2 years ago

What's RAG?

girvo• 2 years ago

Retrieval augmented generation.

Retrieval Augmented Generation (RAG) is a technique where the capabilities of a large language model (LLM) are augmented by retrieving information from other systems and inserting them into the LLM’s context window via a prompt.

(stolen from: https://github.com/psychic-api/rag-stack)

ohmyiv• 2 years ago

Retrieval Augmented Generation. In basic terms, it optimizes output of LLMs by using additional external data sources before answering queries. (That actually might be too basic of a description)

Here:

https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge...

DebtDeflation• 2 years ago

The 10M context ability wipes out most RAG stack complexity immediately

From a technology standpoint, maybe. From an economics standpoint, it seems like it would be quite expensive to jam the entire corpus into every single prompt.

shostack• 2 years ago

Wake me when I can get access without handing over my texts and contacts. I opened the Gemini app on Android and that onerous privacy policy was the first experience. Worse, I didn't seem able to move past accepting giving Google the ability to hoover up my data to disable that in the settings so I just gave up and went back to ChatGPT where I at least generally have control over the data I give it.

tomaskafka• 2 years ago

"I really want to know how they're getting to 10M context, though."

My $5 says it's a RAG or a similar technique (hierarchical RAG comes to mind), just like all other large context LLMs.

cubefox3• 2 years ago

I think Anthropic and OpenAI could also have offered a one million context window a while ago. The relevant architecture breakthrough was probably when a linear increase in context length only required a linear increase in inference compute instead of a quadratic one. Anthropic and then OpenAI achieved linear context compute scaling before an architecture for it was published publicly (MAMBA paper).

bearjaws2• 2 years ago

The problem is, the 128k window performed terribly and showed that attention was mostly limited to the first and last 20%.

Increasing it to 1M just means even more data is ignored.

cubefox3• 2 years ago

Maybe their architecture wasn't as good as MAMBA and Google could use the better architecture thanks to being late to the game...

zippothrowaway• 2 years ago

I've always been suspicious of any announcement from Demis Hassabis since way back in his video game days when he did a monthly article in Edge magainze about the game he was developing. "Infinite Polygons" became a running joke in the industry because of his obvious snake-oil. The game itself, Republic [1], was an uninteresting failure.

He learned how to promote himself from working for Peter "Project Milo" Molyneux and I see similar patterns of hype.

[1] https://en.wikipedia.org/wiki/Republic:_The_Revolution#Marke...

pradn• 2 years ago

The line between delusional and visionary is thin! I know I'm too grounded in "expected value" math to do super outlier stuff like starting a video game company...

Qwero2• 2 years ago

Funny read about his game.

Nonetheless while still underwhelming in comparison to gpt-4 (excluding this announcement as I haven't tried it yet), alpha go, zero and especially fold were tremendous!

COAGULOPATH• 2 years ago

Yeah, it's funny. I used to think "Demis Hassabis...where have I heard that name before?" And then I realized I saw him in the manuals for old Bullfrog games.

rreichman• 2 years ago

And yet - AlphaGo, AlphaZero, AlphaFold...

obblekk• 2 years ago

Very impressive if the benchmarks replicate. Some questions:

* token cost? In multiples of Gemini pro 1

* memory usage? Does already scarce gpu memory become even more of a bottleneck?

* video resolution? Sherlock Jr (1924) is their test video - black and white, 45min, low res

Most curious about the video… I wonder if RAG within video will become the next battlefront

technics2562• 2 years ago

Does anyone actually have access to Ultra yet? It's a lame blog post where it says "it's available!" but the fine print says "by whitelist".

Ok, whatever that means.

OpenAI at least releases it all at once, to everyone.

Szpadel• 2 years ago

oh, openai had a lot of waitlists also, gpt4 API, large context versions etc

sonium• 2 years ago

I just watched the demo with the Apollo 11 transcript. (sidenote: maybe Gemini is named after the space program?).

Wouldn't the transcript or at least a timeline of Apollo 11 be part of the training corpus? So even without the 400 pages in the context window just given the drawing I would assume a prompt like "In the context of Apoll 11, what moment does the drawing refer to?" would yield the same result.

technics2562• 2 years ago

Gemini is named that way because of the collaboration between Google brain and deep mind

singularity20012• 2 years ago

Correct except that it spits out the timestamp

torginus2• 2 years ago

Gemini is named after the spacecraft that put the second person into orbit - pretty aptly named, but not sure if this was the intention.

DrNosferatu2• 2 years ago

Google needs their Apollo.

d0mine2• 2 years ago

The second person was put by MR-3 (Mercury, not Gemini) https://en.m.wikipedia.org/wiki/Timeline_of_space_travel_by_...

empath-nirvana5• 2 years ago

i asked chatgpt4 to identify three humorous moments in the apollo 11 transcript and it hallucinated all 3 of them (i think -- i can't find what it's referring to). Presumably it's in it's corpus, too.

The "Snoopy" Moment: During the mission, the crew had a small, black-and-white cartoon Snoopy doll as a semi-official mascot, representing safety and mission success. At one point, Collins joked about "Snoopy" floating into his view in the spacecraft, which was a light moment reflecting the camaraderie and the use of humor to ease the intense focus required for their mission.

The "Biohazard" Joke: After the successful moon landing and upon preparing for re-entry into Earth's atmosphere, the crew humorously discussed among themselves the potential of being quarantined back on Earth due to unknown lunar pathogens. They joked about the extensive debriefing they'd have to go through and the possibility of being a biohazard. This was a light-hearted take on the serious precautions NASA was taking to prevent the hypothetical contamination of Earth with lunar microbes.

The "Mailbox" Comment: In the midst of their groundbreaking mission, there was an exchange where one of the astronauts joked about expecting to find a mailbox on the Moon, or asking where they should leave a package, playing on the surreal experience of being on the lunar surface, far from the ordinary elements of Earthly life. This comment highlighted the astronauts' ability to find humor in the extraordinary circumstances of their journey.

htrp• 2 years ago

Gemini 1.5 delivers dramatically enhanced performance. It represents a step change in our approach, building upon research and engineering innovations across nearly every part of our foundation model development and infrastructure. This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture.

Looks like they fine tuned across use cases and grabbed the mixtral architecture?

sebzim45003• 2 years ago

There's no way that's all it is, scaling mixtral to a context length of 10M while maintaining any level of reasoning ability would be extremely slow. If the only purpose of the model was to produce this report then maybe that's possible, but if they plan on actually deploying this to end users then there is no way they can run quadratic attention on 10M tokens.

joak3• 2 years ago

«We’ll also introduce 1.5 Pro with a standard 128,000 token context window when the model is ready for a wider release»

So actually they are lagging: their 128k model is yet to be released while OpenAI released theirs some months ago.

joak3• 2 years ago

Their 10M tokens demo is impressive though. They "released" a demo. Confusing...

kyrra• 2 years ago

See: https://blog.google/technology/ai/google-gemini-next-generat...

Gemini 1.5 Pro comes with a standard 128,000 token context window. But starting today, a limited group of developers and enterprise customers can try it with a context window of up to 1 million tokens via AI Studio and Vertex AI in private preview.

joak3• 2 years ago

Gemini 1.5 Pro is not yet released: «Starting today, we’re offering a limited preview of 1.5 Pro to developers and enterprise customers via AI Studio and Vertex AI»

Something like an alpha version.

Limited preview in their jargon.

iamgopal• 2 years ago

AI race is amazing, Nvidia reaping the benefits now, but soon the world.

cubefox3• 2 years ago

The whitepaper says the Buster Keaton film was reduced to 1 FPS before being fed in. Apparently multi-modal language models can only read individual pictures, so videos have to be reduced to a series of frames. I assume animal brains are more efficient than that. E.g. by only feeding the "changes/difference over time" instead of a sequence of time slices.

riku_iki• 2 years ago

it will probably eventually be improved by adding some encoder on top of LLM, which will encode 60 frames into 1 while attempting to preserve information..

freedomben2• 2 years ago

Our teams continue pushing the frontiers of our latest models with safety at the core.

They're not kidding, Gemini (at least what's currently available) is so safe that it's not all that useful.

The "safety" permeates areas where you wouldn't even expect it, like refusing to answer questions about "unsafe" memory management in C. It interjects lectures about safety in answers when you didn't even ask it to do that in the question.

For example, I clicked on one of the four example questions that Gemini proposes to help you get started and it was something like "Write an SMS calling in sick. It's a big presentation day and I'm sad to let the team down." Gemini decided to tell me that it can't impersonate positions of trust like medical professionals or employers (which is not at all what I asking it to do).

The other things I asked it, it gave me wrong and obviously wrong answers. The funniest (though glad it was obviously wrong) was when I asked it "I'm flying from Karachi to Denver. Will I need to pick up my bags in Newark?" and it told me "no, because Karachi to Newark is a domestic flight"

Unless they stop putting "safety at the core," or figure out how to do it in a way that isn't unnecessarily inhibiting, annoying, and frankly insulting (protip: humans don't like to be accused of asking for unethical things, especially when they weren't asking for them. when other humans do that to us, we call that assuming the worst and it's a negative personality trait), any announcements/releases/breakthroughs from Google are going to be a "meh" for me.

dghlsakjg• 2 years ago

This is incredible if it isn't just hype!

I hope the demos aren't fudged/scripted like Google did with Gemini 1.0

amf122• 2 years ago

These demos seem to be videos from AI studio, and which display the time in seconds. Hopefully not fudged.

EZ-E2• 2 years ago

Remember AI Dungeon and how it was frustrating about how it would forget what happened previously? With a 10M context window, am I right to assume it would be possible to weave a story which would span with multiple multiple books worth of content? (more or less 1400 pages)

dougmwne2• 2 years ago

Pretty much! Check out this demo of finding a scene in a 1400 page book based on a stick figure drawing. Mind blowing, right?

https://twitter.com/JeffDean/status/1758148159942091114

EZ-E2• 2 years ago

In theory it would be possible to drop a book and just say "hey Google, create a sequel"

But I doubt it is /that/ good, it's not like we can test it either

VikingCoder• 2 years ago

Dear Google,

Teach Gemini how to be a Dungeon Master, and run free adventures at Comic Con.

Then offer it up as a subscription.

Sincerely,

Everyone

famouswaffles5• 2 years ago

10M tokens is about 25,000 pages. 10M tokens is also never coming to production and is solely research testing.

1M tokens is what they've said will be available for production and is about 2,500 pages.

eigenvalue• 2 years ago

Based on what I've seen so far, I think the probability that this is actually better than GPT4 on the kind of real world coding tasks that I use it for is less than 1%. Literally everything from Google on this has been vaporware or laughably bad in actual practice in my personal experience. Which is totally insane to me given their financial resources, human resources, and multi-year lead in AI/DL research, but that's what seems to have happened. I certainly hope that they can develop and actually release a capable model, but at this point, I think you have to be deeply skeptical of everything they say until such a model is available for real by the public and you can try it on actual, real tasks and not fake benchmark nonsense and waitlists.

scarmig3• 2 years ago

One interesting tidbit from the technical report:

HumanEval is an industry standard open-source evaluation benchmark (Chen et al., 2021), but we found controlling for accidental leakage on webpages and open-source code repositories to be a non-trivial task, even with conservative filtering heuristics. An analysis of the test data leakage of Gemini 1.0 Ultra showed that continued pretraining on a dataset containing even a single epoch of the test split for HumanEval boosted scores from 74.4% to 89.0%, highlighting the danger of data contamination. We found that this sharp increase persisted even when examples were embedded in extraneous formats (e.g. JSON, HTML). We invite researchers assessing coding abilities of these models head-to-head to always maintain a small set of truly held-out test functions that are written in-house, thereby minimizing the risk of leakage. The Natural2Code benchmark, which we announced and used in the evaluation of Gemini 1.0 series of models, was created to fill this gap. It follows the exact same format of HumanEval but with a different set of prompts and tests.

llm_trw• 2 years ago

Yeah. I'll believe that when I can use it.

DidISayTooMuch• 2 years ago

How can I fine tune these models for my use? Their docs isn't clear whether the Gemini models are fine tuneable.

qwertox• 2 years ago

As a sidenote, it's worth clicking the play button and then checking how they're highlighting the current paragraph and word in the inspector.

aubanel2• 2 years ago

For reference, here is the technical report: https://storage.googleapis.com/deepmind-media/gemini/gemini_...

dumbmachine• 2 years ago

It would, probably, be cost prohibitive to use 10M context to it's fullest each time.

I instead hope for to have an api to access to the context as a datastore, so like RAG we can control what to store but unlike rag all data stays within context.

killthebuddha• 2 years ago

10M tokens is an absolute game changer, especially if there's no noticeable decay in quality with prompt size. We're going to see things like entire domain specific languages embedded in prompts. IMO people will start thinking of the prompt itself as a sort of runtime rather than a static input.

Back when OpenAI still supported raw text completion with text-davinci-003 I spent some time experimenting with tiny prompt-embedded DSLs. The results were very, very, interesting IMO. In a lot of ways, text-davinci-003 with embedded functions still feels to me like the "smartest" language model I've ever interacted with.

I'm not sure how close we are to "superintelligence" but for baseline general intelligence we very well could have already made the prerequisite technological breakthroughs.

empath-nirvana5• 2 years ago

It's pretty slow, though looks like up to 60 seconds for some of the answers, and uses god knows how much compute, so there's probably going to be some trade offs -- you're going to want to make sure that that much context is actually useful for what you want.

drusepth• 2 years ago

TBF: when talking about the first "superintelligence", I'd expect it to take unreasonable amounts of compute and/or be slow -- that can always be optimized. Bringing it into existence in the first place is the hardest part.

Yusefmosiah• 2 years ago

I see a lot of talk about retrieval over long context. Some even think this replaces RAG.

I don't care if the model can tell me which page in the book or which code file has a particular concept. RAG already does this. I want the model to notice how a concept is distributed throughout a text, and be able to connect, compare, contrast, synthesize, and understand all the ways that a book touches on a theme, or to rewrite multiple code files in one pass, without introducing bugs.

How does Gemini 1.5's reasoning compare to GPT-4? GPT-4 already has superhuman memory; its bottleneck is its relatively weak reasoning.

sinuhe692• 2 years ago

In my experience (I work mostly and deeply with Bard/Gemini), the reasoning capability of Gemini is quite good. Gemini Pro is already much better than ChatGPT 3.5, but they still make quite a few mistakes along the way. What is more worrying is that when these models made mistakes, they tried really hard to justify their reasoning (errors), practically misleading the users. Because of their high mimicry ability, users really have to pay attention to validate and eventually spot the errors. Of course, this is still far below the human level, so I'm not sure whether they add value or are more of a burden.

famouswaffles5• 2 years ago

The most impressive demonstration of long context is this in my opinion,

https://imgur.com/a/qXcVNOM

Testing language translation abilities of an extremely obscure language after passing in one grammar book as context.

petargyurov• 2 years ago

Version number suggests they're waiting to announce something bigger already?

bloopernova• 2 years ago

Hooray for competition.

luke-stanley• 2 years ago

Still no Ultra model API available to UK devs? Considering Deepmind's London base, this is kinda strange. Maybe they could ask Ultra how to roll it out faster?

bobvanluijt2• 2 years ago

Demo with Google AI Studio: https://twitter.com/bobvanluijt/status/1758185143116730875

processing• 2 years ago

just wade through documentation to access it?

clicking on the AI studio link doesn't show me the app page - it redirects to a document on early access. I do as required - go back and try clicking on the AI studio link and I'm redirected to the document on turning early access.

frustrating.

robertlagrant• 2 years ago

Slightly surprisingly I can't get to AI Studio from the UK. It is available in quite a few countries, but not here.

ChildOfChaos2• 2 years ago

Is this just more nonsense from Google though? I expect big things from Google, but they need to shut up and actually release stuff instead of saying how amazing there stuff is and then release potato ai, nothing they have done in the AI space recently has lived up to any of the hype, they should stay silent for a bit then release something that kills GPT4 if they honestly are able but instead they are just full of hype.

sinuhe692• 2 years ago

Yeah, their Gemini demo was a disaster. But they have released their Ultra model for the general audience, so you can test them yourself. Talking about killing the competitor is a little funny, considering they are all generative LLM based on the same principles (and general architecture) with their inherent flaws and shortcomings. All of them can not even execute a basic plan like a cheap human assistant. So their values are very limited.

Breakthrough will only come with a next generation architecture. LLM for special domains is currently the most promising approach.

ChildOfChaos2• 2 years ago

Yeah but even with ultra they kept saying how it was better than GPT4 and then when it actually got released it was awful.

jeffbee• 2 years ago

A little off-topic I guess, but is anyone else seeing what I am seeing: a total inability to actually upgrade to paid Gemini? Every time I try to sign up it serves me an error page: "We're sorry - Google One storage plans aren't available right now."

DrNosferatu2• 2 years ago

Did they say a general availability date?

(a bit confused)

dang• 2 years ago

There's also https://twitter.com/JeffDean/status/1758146022726041615

(via https://news.ycombinator.com/item?id=39383593, but we merged those comments hither)

topicseed3• 2 years ago

1 million tokens?? This is wild and a lot of RAG can be removed.

topicseed3• 2 years ago

Is this going to be only for consumer Gemini app or for API/Vertex too? The context window is..... Simply lovely.

summerlight3• 2 years ago

One interesting proposal here is a multiple NIAH retrieval benchmark. When they put 100 needles, then the recall rate becomes considerably lower, something around 60~70%. Not sure what's the exact configuration of this benchmark, but intuitively this makes sense and should be a critical metric for the model's reliability.

reissbaker2• 2 years ago

The long context length is of course incredible, but I'm more shocked that the Pro model is now on par with Ultra (~GPT-4, at least the original release). That implies when they release 1.5 Ultra, we'll finally have a GPT-4 killer. And assuming that 1.5 Pro is priced similarly to the current Pro, that's a 4x price advantage per-token.

Not surprising that OpenAI shipped a blog post today about their video generation — I think they're feeling considerable heat right now.

topicseed3• 2 years ago

Gemini 1 Ultra was also said to be on par with ChatGPT 4 and it's not really there so let's see for ourselves when we can get our hands on it.

reissbaker2• 2 years ago

Ultra benchmarked around the original release of GPT-4, not the current model. My understanding is that was fairly accurate — it's close to current GPT-4 but not quite equal. However, close-to-GPT-4 but 4x cheaper and 10x context length would be very impressive and IMO useful.

tigershark2• 2 years ago

Feeling the heat? Did you actually watch the videos? That was a huge leap forward compared to anything existing at the moment. Order of magnitudes away from a blog post discussing a model that maybe will finally be on par with chat gtp 4...

mupuff12342• 2 years ago

The openai announcement is also more or less a blog post, isn't it?

Do we know how much time or money does it take to create a movie clip?

m3kw9• 2 years ago

imagine sending 5-10 mbs over the network per request, and the cost per token. You may accidently go broke after a big lag.

system2• 2 years ago

Let's hope this lowers the pricing of GPT-4 to GPT3.5 levels. Because of Open AI's ridiculous pricing, we can't use it regularly as it would cost us thousands of dollars per month.

stolsvik• 2 years ago

So, this has native image/video modality. I wonder whether that gives it an edge in physical / world understanding? That is, handling and navigating our 3/4 dimensions? Cause and effect and so on?

animanoir• 2 years ago

Google is so finished, they are so late on this.

tmaly• 2 years ago

Is there a $20 a month option for 1.5 Ultra?

If there is, where do I sign up?

ancorevard• 2 years ago

Is this a blog post of did they actually ship?

jstummbillig• 2 years ago

Imagine a day, when your new recording setting 10M token context model is not enough to make it to hn #1

Wild times.

thot_experiment• 2 years ago

I gotta say, I've been trying out Gemini recently and it's embarrassingly bad. I can't take anything google puts out seriously when their current offerings are so so much worse than ChatGPT (or even local llama!).

As a particularly egregious example, yesterday night I gave Gemini a list of drinks and other cocktail ingredients I had laying around and asked for some recommendations for cute drinks that I could make. It's response:

I'm just a language model, so I can't help you with that.

ChatGPT 3.5 came up with several delicious options with clear instructions, but it's not just this instance, I've NEVER gotten a response from Gemini that I even felt was more useful than just a freaking bing search! Much less better than ChatGPT. I'm just going to assume they're using cherrypicked metrics to make themselves feel better until proven otherwise. I have zero confidence in Google's AI plays, and I assume all their competent talent is now at OpenAI or Anthropic.

samspenc• 2 years ago

My experiences are similar, but I think we are talking about the Gemini free model, available on the Google Gemini website. I think the rest of the comments are saying the paid versions (Pro / Ultra) are significantly better, though I haven't tested it myself to compare.

shellfishgene• 2 years ago

I have the 2 months trial for the paid version, and find myself going back to free ChatGPT often. Gemini loves to put everything in bullet point lists and short paragraphs with subheadings for example, even when asking for a letter. I'm not a heavy user, but it seems to not quite get what I want often. Not important but annoying: It starts almost every answer with "Absolutely!", even when it doesn't match the question (e.g. "How does x work?").

staticman23• 2 years ago

I don't think "I'm just a language model, I can't help you with that" comes from Gemini. Google has a seperate censorship model that blocks you from receiving Gemini's response in certain situations.

When Gemeni (Ultra) refuses to do something itself it is more verbose and specific as to why it won't do it, it my experience.

mikeweiss2• 2 years ago

Does anyone know what kinds of GPUs/Chips Google is using for Gemini? They aren't using Nvidia correct?

sackfield• 2 years ago

TPUs: https://cloud.google.com/tpu?hl=en

mikeweiss2• 2 years ago

So Google doesn't rely on Nvidia at all? How come they are the only ones that can manage to use non Nvidia chips and compete with Open AI?

jakub_g• 2 years ago

Very off-topic but I can't help, the pace of change reminds of the "Bates 4000" sketch from The Onion Movie:

https://m.youtube.com/watch?v=fw7FniaeaSo

TaylorAlexander3• 2 years ago

Does anyone know how to get Gemini to help refactor code? I’m trying to paste in my code file and the web page says “an error has occurred” and the code does not show up in the code window. I tried signing up for Gemini Advanced and that didn’t help. I also tried pointing it to the file on GitHub and it said it couldn’t access it. People here are saying Gemini is great for code refactoring. How do you do that?

noisy_boy• 2 years ago

I feel sad for those who are in law school right now.

arthur_sav• 2 years ago

zzZZzzzZ

ylluminate• 2 years ago

Has anyone given an idea of the release timeline for 1.5?

newzisforsukas3• 2 years ago

ah, so google found a moat.

Aeolun4• 2 years ago

This got my trying Gemini, but doing so is such a hassle that I'm almost ready to give up. Trying out ChatGPT is as simple as signing up (either for pro, or the API), and getting a single API key.

Google requires me to navigate their absolutely insane console (seriously, I thought the AWS console was bad, but GCP takes the cake), only to tell me there is not even a way to get an API key... I had to ask Gemini through the built in interface to figure that out.

fakedang2• 2 years ago

API keys are fairly straightforward in GCP though - there's an entire section for that, and even if you're stuck, the search console works.

Aeolun4• 2 years ago

Here is an entire comment complaining it’s not straightforward, and you make a comment that essentially comes down to ‘no it is’?

is_true3• 2 years ago

https://aistudio.google.com/

Unfortunately there's a waitlist for the 1.5 architecture

jiggawatts2• 2 years ago

That just redirects to an FAQ page with instructions like "If you get X error, then..."

There is no error, it just redirects me.

Fail.

Aeolun4• 2 years ago

That, and I found somewhere it wants to use all my data for training, which was a non-starter. Unfortunately I couldn’t find that page any more after the first time I found it.

needlesslygrim4• 2 years ago

Personally, I've given up on Gemini, as it seems to have been censored to the point of uselessness. I asked it yesterday [0] about C++ 20 Concepts, and it refused to give actual code because I'm under 18 (I'm 17, and AFAIK that's what the age on my Google account is set to). I just checked again, and it gave a similar answer [1]. When I tried ChatGPT 3.5, it did give an answer, although it was a little confused, and the code wasn't completely correct.

This seems to be a common experience, as apparently it refuses to give advice on copying memory in C# [2], and I tried to do what was suggested in this comment [3], but by the next prompt it was refusing again, so I had to stick to ChatGPT.

[0] https://g.co/gemini/share/238032386438

[1] https://g.co/gemini/share/6880989ddfaf

[2] https://news.ycombinator.com/item?id=39312896

[3] https://news.ycombinator.com/item?id=39313567

wrasee2• 2 years ago

From your first link [0]

Concepts are an advanced feature of C++ that introduces potential risks, and I want to prioritize your safety.

Brilliant.

GaryNumanVevo• 2 years ago

I'm with Gemini on this one, 17 years old is too young to be learning about unsafe C++ features, best stick with a memory safe language until you're old enough

needlesslygrim4• 2 years ago

Luckily I only really muck about with C++, and mainly use Rust, so my childlike brain is protected :^)

AaronFriel3• 2 years ago

If you can dig in further - prompt engineer out - the prompt for minors that would be fascinating to report out.

needlesslygrim4• 2 years ago

I'll see what I can do, but I don't expect I'll achieve much.

Breza4• 2 years ago

I've gotten similar pushback. I asked for a solution involving curl and was told that that was not something Gemini could do. Then I clicked the button to see the other drafts of Gemini's responses and got two versions that worked.

shantnutiwari• 2 years ago

No faked videos this time?

I find it hard to trust google nowadays.

_heimdall4• 2 years ago

These announcements always make it clear how little companies releasing new AI models actually care about the risks and concerns of developing artificial intelligence.

CEOs love to talk about how important regulation is, how their company needs to develop it before the "wrong people" do, and how they are concerned about what could happen if AI development goes wrong.

Then they announce the latest model that is aimed at expanding both the accuracy and breadth of use cases across multiple modalities. Sure the release links to a security and ethics page, but that page reads more like a company's internal "pillars of success" document with vague phrases that define little to nothing in the way of real, specific concerns or measures to mitigate them. It basically boils down to "Don't be evil" with no clear definition of what that would mean or how they prevent the new, more powerful and broad reaching system from being used in ways that are "evil".

firtoz• 2 years ago

I don't see any other way, personally.

We can argue that a lot of people have done pretty bad things using the internet, but should it have been regulated in advance?

_heimdall4• 2 years ago

Regulation doesn't have to be the answer though. The very same people talking out of both sides of their mouths here are the ones who can choose to just not invest in it.

Lock up the hardware in an offline facility and experiment there, if they really think it's important. Hell, even just skipping the double speak would be a big step. If they really aren't concerned with the risk then own it, don't tell me its risky while also releasing a new, more powerful version every 6-12 months.

btbuildem• 2 years ago

CEOs are the "wrong people". Leaders of obscenely large organizations, unchecked by law nor ethics, wielding and gatekeeping what amounts to superpowers. They are the literal supervillains, not some shadowy "terrorists" or another made up bogeyman.

_heimdall4• 2 years ago

I'd argue we could get a long way by removing legal protections and incentives for such large organizations. Those protections and incentives seem to me to be the root cause behind such large centralizations of power.

If companies and their leadership couldn't operate so unchecked by our existing laws and public opinion we may not have executives worth worrying about.

For example, if taxes were so easy to dodge and if the public actually had a chance to sue large corporations for damages they may not get so large. If, when losing a lawsuit, companies couldn't shuffle around funds and spin off dummy companies to dodge the pain, and if they weren't often forced to pay pennies on the dollar for lost suits, they may think twice about doing some things. When you know your entire business is actually on the line you have to be more careful.

Throw in election and lobbying reform and we could at least be having a much different conversation about corporate power.

d0mine2• 2 years ago

It has been lobotomized enough already. Look at C++20 Concepts example https://news.ycombinator.com/item?id=39395020

_heimdall4• 2 years ago

Limiting features in the public API aren't quite the same as limiting the technical readability of developing an artificial intelligence or consciousness.

Limiting public features will help a bit with concerns over how someone might use a public GPT API, but the technology advancements will be made either way and ultimately companies won't be able to gate keep who can use it with 100% accuracy. The boom for GPU hardware similarly is pushing us further down the road to AI development and all the moral and ethical questions that go along with it, even if AI companies were to keep use of their GPUs and models private entirely.

nycdatasci• 2 years ago

This includes making Gemini 1.5 more efficient to train and serve, with a new Mixture-of-Experts (MoE) architecture.

Wait. They are abandoning PaLM 2, which was just announced 9 months ago?

gnrlst• 2 years ago

4 weeks ago: Gemini Advanced with Ultra 1.0 Now: Gemini 1.5 Pro

Is it just me or is their branding all over the place.

Our next-generation model: Gemini 1.5

Related Stories