Gemini 2.0: our new AI model for the agentic era

https://blog.google/technology/google-deepmind/google-gemini-ai-update-december-2024/

by meetpateltech • 2 years ago

1015 490 2 years ago

The Gemini 2 models support native audio and image generation but the latter won't be generally available till January. Really excited for that as well as 4o's image generation (whenever that comes out). Steerability has lagged behind aesthetics in image generation for a while now and it's be great to see a big advance in that.

Also a whole lot of computer vision tasks (via LLMs) could be unlocked with this. Think Inpainting, Style Transfer, Text Editing in the wild, Segmentation, Edge detection etc

They have a demo: https://www.youtube.com/watch?v=7RqFLp0TqV0

jncfhnb2• 2 years ago

These are not computer vision tasks…

Jabrov• 2 years ago

What are they, then…?

newfocogi• 2 years ago

Maybe some of these tasks are arguably not aligned with the traditional applications of CV, but Segmentation and Edge detection are definitely computer vision in every definition I've come across - before and after NNs took over.

kthartic2• 2 years ago

I asked Gemini 2.0 Flash (with my voice) whether it natively understands audio or is converting my voice to text. It replied:

"That's an insightful question. My understanding of your speech involves a pipeline first. Your voice is converted to text and then I process the text to understand what you're saying. So I don't understand your voice directly but rather through a text representation of it."

Unsure if this is a hallucination, but is disappointing if true.

Edit: Looking at the video you linked, they say "native audio output", so I assume this means the input isn't native? :(

famouswaffles4• 2 years ago

Native audio output won't be in general availability until early next year.

If you're using Gemini in aistudio(not sure about the real-time API but everything else) then it has native audio input

jerpint• 2 years ago

We’re definitely going to need better benchmarks for agentic tasks, and not just code reasoning. Things that are needlessly painful that humans go through all the time

AuthConnectFail• 2 years ago

it's insane on lmarena for a size, livebench should have it soon too I guess

maeil5• 2 years ago

The size isn't stated, not necessarily a given that it's as small as 1.5-Flash.

bradhilton• 2 years ago

Beats Gemini 1.5 Pro at all but two of the listed benchmarks. Google DeepMind is starting to get their bearings in the LLM era. These are the minds behind AlphaGo/Zero/Fold. They control their own hardware destiny with TPUs. Bullish.

p1esk3• 2 years ago

Are these benchmarks still meaningful?

maeil5• 2 years ago

No, and they haven't been for at least half a year. Utterly optimized for by the providers. Nowadays if a model would be SotA for general use but not #1 on any of these benchmarks, I doubt they'd even release it.

CamperBob23• 2 years ago

I've started keeping an eye out for original brainteasers, just for that reason. GCHQ's Christmas puzzle just came out [1], and o1-pro got 6 out of 7 of them right. It took about 20 minutes in total.

I wasn't going to bother trying those because I was pretty sure it wouldn't get any of them, but decided to give it an easy one (#4) and was impressed at the CoT.

Meanwhile, Google's newest 2.0 Flash model went 0 for 7.

1: https://metro.co.uk/2024/12/11/gchq-christmas-puzzle-2024-re...

dagmx12• 2 years ago

Regarding TPU’s, sure for the stuff that’s running on the cloud.

However their on device TPUs lag behind the competition and Google still seem to struggle to move significant parts of Gemini to run on device as a result.

Of course, Gemini is provided as a subscription service as well so perhaps they’re not incentivized to move things locally.

I am curious if they’ll introduce something like Apple’s private cloud compute.

whimsicalism12• 2 years ago

i don’t think they need to win the on device market.

we need to separate inference and training - the real winners are those who have the training compute. you can always have other companies help with inference

mupuff12344• 2 years ago

Majority of people want better performance, running locally is just a nice to have feature.

YetAnotherNick5• 2 years ago

If the model weights is not open, you can't run it on device anyways.

JeremyNT• 2 years ago

Yeah they've been slow to release end-user facing stuff but it's obvious that they're just grinding away internally.

They've ceded the fast mover advantage, but with a massive installed base of Android devices, a team of experts who basically created the entire field, a huge hardware presence (that THEY own), massive legal expertise, existing content deals, and a suite of vertically integrated services, I feel like the game is theirs to lose at this point.

The only caution is regulation / anti-trust action, but with a Trump administration that seems far less likely.

VirusNewbie• 2 years ago

If you look at where talent is going, it's Anthropic that is the real competitor to Google, not OpenAI.

echelon3• 2 years ago

Gemini in search is answering so many of my search questions wrong.

If I ask natural language yes/no questions, Gemini sometimes tells me outright lies with confidence.

It also presents information as authoritative - locations, science facts, corporate ownership, geography - even when it's pure hallucination.

Right at the top of Google search.

edit:

I can't find the most obnoxious offending queries, but here was one I performed today: "how many islands does georgia have?".

Compare that with "how many islands does georgia have? Skidaway Island".

This is an extremely mild case, but I've seen some wildly wrong results, where Google has claimed companies were founded in the wrong states, that towns were located in the wrong states, etc.

sib301• 2 years ago

This has happened to me zero times. :shrug:

airstrike5• 2 years ago

Doesn't match my experience. It also feels like it's getting better over time.

nilayj• 2 years ago

can you provide some example queries that Gemini in search gets wrong?

echelon3• 2 years ago

I just found another one:

A depsipeptide is a cyclic peptide where one or more amide groups are replaced by ester groups.

Depsipeptides are not necessarily cyclic, and I'd probably use "bond" instead of "group".

https://imgur.com/a/YslvJO2

These errors are happening all the time.

nice__two• 2 years ago

Gemini 1.5 indeed is a lot of hit-and-miss. Also, the politically correct and medical info filtering is limiting its usefulness a lot, IMHO.

I also miss that it’s not yet really as context aware as ChatGPTo4. Even just asking a follow-up question, confuses Gemini 1.5.

Hope Gemini 2.0 will improve that!

adultSwim• 2 years ago

I've found these results quite useful

jonomacd2• 2 years ago

At first, this was true but now it has gotten pretty good. The times it gets things wrong are often not the models fault and just google searches fault.

zb32• 2 years ago

Is this the gemini-exp model on LMArena?

jasonjmcghee• 2 years ago

Both are available on aistudio so I don't think so.

In my own testing "exp 1206" is significantly better than Gemini 2.

Feels like haiku 3.5 vs sonnet 3.5 kind of thing.

warkdarrior• 2 years ago

Yes, LMArena shows Gemini-2.0-Flash-Exp ranking 3rd right now, after Gemini-Exp-1206 and ChatGPT-4o-latest_(2024-11-20), and ahead of o1-preview and o1-mini:

https://lmarena.ai/?leaderboard

zb32• 2 years ago

There's also the "gremlin" model (not reachable directly) and it seems to be pretty smart.. maybe that's the deep research mode?

EDIT: probably not deep research.. is it Google testing their equivalent of o1? who knows..

usaar333• 2 years ago

It looks like gemini-exp-1121 slightly upgraded. 1206 is something else.

crowcroft2• 2 years ago

Big companies can be slow to pivot, and Google has been famously bad at getting people aligned and driving in one direction.

But, once they do get moving in the right direction the can achieve things that smaller companies can't. Google has an insane amount of talent in this space, and seems to be getting the right results from that now.

Remains to be seen how well they will be able to productize and market, but hard to deny that their LLM models aren't really, really good though.

manishsharan3• 2 years ago

> hard to deny that their LLM models aren't really, really good though.

The context window of Gemini 1.5 pro is incredibly large and it retains the memory of things in the middle of the window well. It is quite a game changer for RAG applications.

KaoruAoiShiho• 2 years ago

It looks like long context degraded from 1.5 to 2.0 according to the 2.0 launch benchmarks.

caeril• 2 years ago

Bear in mind that a "1 million token" context window isn't actually that. You're being sold a sparse attention model, which is guaranteed to drop critical context. Google TPUs aren't running inference on a TERABYTE of fp8 query-key inputs, let alone TWO of fp16.

Google's marketing wins again, I guess.

pelorat• 2 years ago

Well, compared to github copilot (paid), I think Gemini Free is actually better at writing non-archaic code.

rafaelmn• 2 years ago

Using Claude 3.5 sonnet ?

jacooper2• 2 years ago

Gemini is coming to copilot soon anyway.

talldayo3• 2 years ago

BERT and Gemma 2B were both some of the highest-performing edge models of their time. Google does really well - in terms of pushing efficiency in the community they're second to none. They also don't need to rely on inordinate amounts of compute because Google's differentiating factor is the products they own and how they integrate it. OpenAI is API-minded, Google is laser-focused on the big-picture experience.

For example; those little AI-generated YouTube summaries that have been rolling out are wonderful. They don't require heavyweight LLMs to generate, and can create pretty effective summaries using nothing but a transcript. It's not only more useful than the other AI "features" I interact with regularly, it doesn't demand AGI or chain-of-thought.

closewith• 2 years ago

Google is laser-focused on the big-picture experience.

This doesn't match my experience of any Google product.

panabee• 2 years ago

With many research areas converging to comparable levels, the most critical piece is arguably vertical integration and forgoing the Nvidia tax.

They haven't wielded this advantage as powerfully as possible, but changes here could signal how committed they are to slaying the search cash cow.

Nadella deservedly earned acclaim for transitioning Microsoft from the Windows era to cloud and mobile.

It will be far more impressive if Google can defy the odds and conquer the innovator's dilemma with search.

Regardless, congratulations to Google on an amazing release and pushing the frontiers of innovation.

crowcroft2• 2 years ago

They need an iPod to iPhone like transition. If they can pull it off it will be incredible for the business.

bloomingkales• 2 years ago

They have to not get blind sided by Sora, while at the same time fighting the cloud war against MS/Amazon.

Weirdly Google is THE AI play. If AI is not set to change everything and truly is a hype cycle, then Google stock withstands and grows. If AI is the real deal, then Google still withstands due to how much bigger the pie will get.

TacticalCoder3• 2 years ago

Nadella deservedly earned acclaim for transitioning Microsoft from the Windows era to cloud and mobile.

You mean by shifting away from Windows for mobile and focusing on iOS and Android?

bushbaba3• 2 years ago

Yet, google continues to show it'll deprecate it's APIs, Services, and Functionality at the detriment of your own business. I'm not sure enterprises will trust Google's LLM over the alternatives. Too many have been burned throughout the years, including GCP customers.

The fact GCP needs to have this page, and these lists are not 100% comprehensive is telling enough. https://cloud.google.com/compute/docs/deprecations https://cloud.google.com/chronicle/docs/deprecations https://developers.google.com/maps/deprecations

Steve Yegge rightfully called this out, and yet no change has been made. https://medium.com/@steve.yegge/dear-google-cloud-your-depre...

weatherlite6• 2 years ago

GCP grew 35% last quarter , just saying ...

verdverm2• 2 years ago

At least GCP makes them easy to find

Some guy had to do it for Azure, then he went to work for them and it is now deprecated itself

https://blog.tomkerkhove.be/2023/03/29/sunsetting-azure-depr...

StableAlkyne• 2 years ago

Remains to be seen how well they will be able to productize and market

The challenge is trust.

Google is one of the leaders in AI and are home to incredibly talented developers. But they also have an incredibly bad track record of supporting their products.

It's hard to justify committing developers and money to a product when there's a good chance you'll just have to pivot again once they get bored. Say what you will about Microsoft, but at least I can rely on their obsession with supporting outdated products.

egeozcan3• 2 years ago

they also have an incredibly bad track record of supporting their products

Incredibly bad track record of supporting products that don't grow. I'm not saying this to defend Google, I'm still (perhaps unreasonably) angry because of Reader, it's just that there is a pattern and AI isn't likely to fit that for a long while.

TIPSIO2• 2 years ago

Yes. Imagine Google banning your entire Google account / Gmail because you violated their gray area AI terms ([1] or [2]). Or, one of your users did via an app you made using an API key and their models.

With that being said, I am extremely bullish on Google AI for a long time. I imagine they land at being the best and cheapest for the foreseeable future.

[1] https://policies.google.com/terms/generative-ai

[2] https://policies.google.com/terms/generative-ai/use-policy

fluoridation10• 2 years ago

Say what you will about Microsoft, but at least I can rely on their obsession with supporting outdated products.

Eh... I don't know about that. Their tech graveyard isn't as populous as Google's, but it's hardly empty. A few that come to mind: ATL, MFC, Silverlight, UWP.

dotancohen2• 2 years ago

  > Google is one of the leaders in AI and are home to incredibly talented developers. But they also have an incredibly bad track record of supporting their products.

This is why we've stayed with Anthropic. Every single person I work with on my current project is sore at Google for discontinuing one product or another - and not a single one of them mentioned Reader.

We do run some non-customer facing assets in Google Cloud. But the website and API are on AWS.

bastardoperator• 2 years ago

Putting your trust in Google is a fools errand. I don't know anyone that doesn't have a story.

TacticalCoder3• 2 years ago

But they also have an incredibly bad track record of supporting their products.

I don't know about that: my wife built her first SME on Google Workspace / GSuite / Google Apps for domain (this thing changed names so many times I lost track). She's now running her second company on Google tools, again.

All she needs is a browser. At one point I switched her from Windows to OS X. Then from OS X to Ubuntu.

Now I just installed Debian GNU/Linux on her desktop: she fires up a browser and opens up Google's GMail / GSuite / spreadsheets and does everything from there.

She's a happy paying customer of Google products since a great many years and there's actually phone support for paying customers.

I honestly don't have many bad things to say. It works fine. 2FA is top notch.

It's a much better experience than being stuck in the Windows "Updating... 35%" "here's an ad on your taskbar" "you're computer is now slow for no reason" world.

I don't think they'l pull the plug on GSuite: it's powering millions and millions of paying SMEs around the world.

boringg• 2 years ago

Can I add they have a bad track record of supporting new products. Gmail, google, gsuite seem to be well supported.

RevertBusload• 2 years ago

Surface Duo would like to have a word

crazygringo• 2 years ago

and Google has been famously bad at getting people aligned and driving in one direction.

To be fair, it's not that they're bad at it -- it's that they generally have an explicit philosophy against it. It's a choice.

Google management doesn't want to "pick winners". It prefers to let multiple products (like messaging apps, famously) compete and let the market decide. According to this way of thinking, you come out ahead in the long run because you increase your chances of having the winning product.

Gemini is a great example of when they do choose to focus on a single strategy, however. Cloud was another great example.

xnx5• 2 years ago

I definitely agree that multiple competing products is a deliberate choice, but it was foolish to pursue it for so long in a space like messaging apps that has network effects.

As a user I always still wish that there were fewer apps with the best features of both. Google's 2(!) apps for AI podcasts being a recent example : https://notebooklm.google.com/ and https://illuminate.google.com/home

tbarbugli• 2 years ago

Google is not winning on cloud, AWS is winning and MS gaining ground.

bwb• 2 years ago

So far, for my tests, it has performed terribly compared to ChatGPT and Claude. I hope this version is better.

aerhardt2• 2 years ago

seems to be getting the right results
hard to deny that their LLM models aren't really, really good though

I'm so scarred by how much their first Gemini releases sucked that the thought of trying it again doesn't even cross my mind.

Are you telling us you're buying this press release wholesale, or you've tried the tech they're talking about and love it, or you have some additional knowledge not immediately evident here? Because it's not clear from your comment where you are getting that their LLM models are really good.

MaxDPS• 2 years ago

I’ve been using Gemini 1.5 Pro for coding and it’s been great.

TacticalCoder3• 2 years ago

but hard to deny that their LLM models aren't really, really good though

Although I do still pay for ChatGPT, I find it dog slow. ChatGPT is simply way too slow to generate answers. It feels like --even though of course it's not doing the same thing-- I'm back to the 80s with my 8-bit computer printing thing line by line.

Gemini OTOH doesn't feel like that: answers are super fast.

To me low latency is going to be the killer feature. People won't keep paying for models that are dog slow to answer.

I'll probably be cancelling my ChatGPT subscription soon.

sky22245• 2 years ago

Maybe I've just hit a streak of good outputs, but I've also been noticing the automatic gemini search when doing google searches to have been significantly more useful than it was previously.

About a year ago, I was saying that Google was potentially walking toward its own grave due to not having any pivots that rivaled OpenAI. Now, I'm starting to think they've found the first few steps toward an incredible stride.

jncfhnb2• 2 years ago

Am I alone in thinking the word “agentic” is dumb as shit?

Most of these things seem to just be a system prompt and a tool that get invoked as part of a pipeline. They’re hardly “agents”.

They’re modules.

thomassmith65• 2 years ago

It's easier for consultants and sales people to sell to enterprise if the terminology is familiar but mysterious.

Bad

  1. installed Antivirus software
  2. added screen-size CSS rules
  3. copied 'Assets' harddrive to DropBox
  4. edited homepage to include Bitcoin wallet address link
  5. upgraded to ChatGPT Pro

"Good"

  1. Cyber-security defenses
  2. Responsive Design implementation
  3. Cloud Storage
  4. Blockchain Technology gateway
  5. Agentic enhancements

endorphine• 2 years ago

George Carlin's "Euphimisms" comes to mind.

kthartic2• 2 years ago

These are great examples. Also an excellent technique for relaying project updates to non-technical managers

xnx5• 2 years ago

Controlling a browser in Project Mariner seems very agentic: https://youtu.be/Fs0t6SdODd8?t=86

uludag• 2 years ago

Definitely not alone. With all the this money at stake, coining dumb terms like this might make you a pretty penny.

It's like a meme that can be milked for monetization.

Agentus• 2 years ago

The beauty of LLMs isn’t just these coding objects speak human vernacular but they can be concatenated with human vernacular prompts and that itself can be used as an input, command or output sensibly without necessarily causing error even if a series of inputs combinations weren't preprogrammed.

I have an A.I. textbook that has agent terminology that was written preLLm days. agents are just autonomous ish code that loops on itself with some extra functionality. LLMs in their elegance can more easily out the box selfloop just on the basis concatenating language prompts, sensibly. They are almost agent ready out the box by this very elegant quality(the textbook agentic diagram is just a conceptual self perpetuation loop), except…

Except they fail at a lot or get stuck at hiccups. But, here is a novel thought. What if an LLM becomes more agentic (ie more able to sustain autonomous chain prompts that do actions without a terminal failure) and less copilotee not by more complex controlling wrapper self perpetuation code, but by means of training the core llm itself to more fluidly function in agentic scenarios.

a better agentically performing llm that isnt mislabeled with a bad buzzword might not reveal itself in its wrapper control code but through it just performing better in an typical agentic loop or environment conditions with whatever initiating prompt, control wrapper code, or pipeline that initiates its self perpetuation cycle.

WA• 2 years ago

Gemini, too, for the sole reason that non-native speakers have no clue how to pronounce it.

kaashif• 2 years ago

Also, people at NASA pronounce it two ways, even native speakers of English.

purple-leafy• 2 years ago

pronounced: juh-meany .... right?

Havoc• 2 years ago

“agentic” is dumb as shit?

It'll create endless consulting opportunities for projects that never go anywhere and add nothing of value unless you value rich consultants.

sippeangelo• 2 years ago

Aside from sounding "dumb as shit", I also think it's the completely wrong abstraction. Every time I've split up work between multiple specialized LLM "agents", they've done an WAY worse job than one monolith LLM that had access to all the tools it needs and can "reason" about the whole task.

All the common tricks, like creating a list of steps that are then executed by specialized agents in order, for example, fall flat as soon as one agent returns a result that contradicts the initial steps. It's simply a bandaid for short context sizes and LLMs that can't remain focused past the first few thousand tokens of prompt.

brokensegue2• 2 years ago

Any word on price? I can't find it at https://ai.google.dev/pricing

Oras• 2 years ago

£18/month

https://gemini.google/advanced/?Btc=web&Atc=owned&ztc=gemini...

then sign in with Google account and you'll see it

brokensegue2• 2 years ago

Oh but I only care about api pricing

gman83• 2 years ago

I've been using Gemini Flash for free through the API using Cline for VS Code. I switch between Claude and Gemini Flash, using Claude for more complicated tasks. Hope that the 2.0 model comes closer to Claude for coding.

SV_BubbleTime3• 2 years ago

Or… just continue using Claude?

serjester2• 2 years ago

Agreed - tried some sample prompts on our data and the rough vibe check is that flash is now as good as the old pro. If they keep pricing the same, this would be really promising.

airstrike5• 2 years ago

OT: I’m not entirely sure why, but "agentic" sets my teeth on edge. I don't mind the concept, but the word itself has that hollow, buzzwordy flavor I associate with overblown LinkedIn jargon, particularly as it is not actually in the dictionary...unlike perfectly serviceable entries such as "versatile", "multifaceted" or "autonomous"

geodel3• 2 years ago

Huh, all three words you mentioned as replacement are equally buzzwordy and I see them a lot in CVs while screen candidates for job interview.

airstrike5• 2 years ago

At least all three of them are actually in the dictionary

raincole• 2 years ago

Versatile implies it can to more kinds of tasks (than it's predecessor or competitor). Agentic implies it requires less human intervention.

I don't think these are necessary buzzwords if the product really does what they imply.

lolinder• 2 years ago

They agree—they're saying that at least those buzzwords are in the dictionary, not that they'd be a good replacement for "agentic".

thom• 2 years ago

I'm personally very glad that the word has adhered itself to a bunch of AI stuff, because people had started talking about "living more agentically" which I found much more aggravating. Now if anyone states that out loud you immediately picture them walking into doors and misunderstanding simple questions, so it will hopefully die out.

OutOfHere• 2 years ago

To play devil's advocate, the correct use of the word would be when multiple AIs are coordinating and handing off tasks to each other with limited context, such that the handoffs are dynamically decided at runtime by the AI, not by any routine code. I have yet to see a single example where this is required. Most problems can be solved with static workflows and simple rule based code. As such, I do believe that >95% of the usage of the word is marketing nonsense.

maeil5• 2 years ago

I actually have built such a tool (two AIs, each with different capabilities), but still cringe at calling at agentic. Might just be an instinctive reflex.

danpalmer3• 2 years ago

I think this sort of usage is already happening, but perhaps in the internal details or uninteresting parts, such as content moderation. Most good LLM products are in fact using many LLM calls under the hood, and I would expect that results from one are influencing which others get used.

jasonsteving• 2 years ago

You nailed an interesting nuance there about agents needing to make their own decisions!

I'm getting fairly excited about "agentic" solutions to the point that I even went out of my way to build "AgentOfCode" (https://github.com/JasonSteving99/agent-of-code) to automate solving Advent of Code puzzles by iteratively debugging executions of generated unit tests (intentionally not competing on the global leaderboard).

And even for this, there's actually only a SINGLE place in the whole "agent" where the models themselves actually make a "decision" on what step to take next, and that's simply deciding whether to refactor the generated unit tests or the generated solution based on the given error message from a prior failure.

m3kw93• 2 years ago

Yeah I hate it when AI companies throw around words like AGI and agentic capabilities. It’s non sense to most people and ambiguous at best

christianqchung• 2 years ago

This is what other replies are missing - I've been following AI closely since GPT 2 and it's not immediately clear what agentic means, so to other people, the term must be even less clear. Using the word autonomous can't be worse than agentic imo.

ramoz4• 2 years ago

Need a general term for autonomous intelligent decision making.

airstrike5• 2 years ago

Isn't that just "intelligent"?

aithrowawaycomm• 2 years ago

No, we need a scientific understanding of autonomous intelligent decision-making. The problem with “agentic AI” is the same old “Artificial Intelligence, Natural Stupidity” problem: we have no clue what “reasoning” or “intelligence” or “autonomous” actually means in animals, and trying to apply these terms to AI without understanding them (or inventing a new term without nailing down the underlying concept) is doomed to fail.

wepple• 2 years ago

Versatile is far worse. It’s so broad to the point of meaninglessness. My garden rake is fairly versatile.

Agentic to me means that it acts somewhat under its own authority rather than a single call to an LLM. It has a small degree of agency.

heresie-dabord• 2 years ago

Take good care of your teeth, my friend. It's a new era in agentic factuality.

dsr_• 2 years ago

agentic obviously means "not gentic" and gent is from the Latin for "people".

agentic == not people.

Quite sensible, really.

geodel3• 2 years ago

Just searched for GVP vs SVP and got:

"GVP stands for Good Pharmacovigilance Practice, which is a set of guidelines for monitoring the safety of drugs. SVP stands for Senior Vice President, which is a role in a company that focuses on a specific area of operations."

Seems lot of pharma regulation in my telecom company.

PaulWaldman• 2 years ago

Anecdotally, using the Gemini App with "Gemini Advanced 2.0 Flash Experimental", the response quality is ignorantly improved and faster at some basic Python and C# generation.

xnx5• 2 years ago

ignorantly improved

autocorrect of "significantly improved"?

oldpersonintx• 2 years ago

the demos are amazing

I need to rewire my brain for the power of these tools

this plus the quantum stuff...Google is on a win streak

SubiculumCode• 2 years ago

Considering so many of us would like more vRAM than NVIDIA is giving us for home compute, is there any future where these Trillium TPUs become commodity hardware?

geodel3• 2 years ago

So many of us are probably in thousands they need to be 3 order magnitude higher before Google can even think of it.

kajecounterhack• 2 years ago

Power concerns aside, individual chips in a TPU pod don't actually have a ton of vRAM; they rely on fast interconnects between a lot of chips to aggregate vRAM and then rely on pipeline / tensor parallelism. It doesn't make sense to try to sell the hardware -- it's operationally expensive. By keeping it in house Google only has to support the OS/hardware in their datacenter and they can and do commercialize through hosted services.

Why do you want the hardware vs just using it in the cloud? If you're training huge models you probably don't also keep all your data on prem, but on GCS or S3 right? It'd be more efficient to use training resources close to your data. I guess inference on huge models? Still isn't just using a hosted API simpler / what everyone is doing now?

katamari-damacy• 2 years ago

Is it better than GPT4o? Does it have an API?

jerrygenser• 2 years ago

API is accessible via Vertex AI on Google Cloud in preview. I think it's also available in the consumer Gemini Chat.

nmfisher• 2 years ago

Also FYI for anyone else, I can't actually see the gemini-2.0-flash-exp list in the model directory for Vertex AI in GCP Console, but passing in google/gemini-2.0-flash-exp to the vertexai Python SDK does actually work.

(you'll probably also need to increase your quotas right away, the default is only 10 requests per minute).

kgwgk• 2 years ago

https://ai.google.dev/gemini-api/docs/models/gemini-v2

chrsw• 2 years ago

Instead of throwing up tables of benchmarks just let me try to do stuff and see if it's useful.

topicseed2• 2 years ago

Is it on AI studio already?

jonomacd2• 2 years ago

Yes it is. Including the live features. It is pretty impressive. Basically voice mode with a live video feed as well.

topicseed2• 2 years ago

Just played with it and it's great! A good 2.0 release I think.

siliconc0w2• 2 years ago

What's everyone's favorite LLM leaderboard? Gemini 2 seems to be edging out 4o on chatbot arena(https://lmarena.ai/?leaderboard)

SV_BubbleTime3• 2 years ago

AI benchmarks and leaderboards are complete nonsense though.

Find something you like, use it, be ready to look again in a month or two.

falcor84• 2 years ago

With the accelerating progress, the "be ready to look again" is becoming a full time job that we need to be able to delegate in some way, and I haven't found anything better than benchmarks, leaderboards and reviews.

EDIT: Typo

siliconc0w2• 2 years ago

FWIW I've found the 'coding' 'category' of the leaderboard to be reasonably accurate. Claude was the best, o1-mini then was typically stronger, now the Gemini Exp 1206 is at the top.

I find myself just paying a la carte via the API rather than paying the $20/mo so I can switch between the models.

hombre_fatal3• 2 years ago

poe.com has a decent model where you buy credits and spend them talking to any LLM which makes it nice to swap between them even during the same conversation instead of paying for multiple subscriptions.

Though gpt-4o could say "David Mayer" on poe.com but not on chat.openai.com which makes me wonder if they sometimes cheat and sneak in different models.

manishsharan3• 2 years ago

Leaderboards are not that useful for measuring real-life effectiveness of the models atleast in my day-today usage.

I am currently struggling to diagnose an ipv6 mis-configuration in my enormous aws cloudformation yaml code. I gave the same input to Claude Opus, Gemini and ChatGPT ( o1 and 4o).

4o was the worst. verbose and waste of my time.

Claude completely went off-tangent and began recommending fixes for ipv4 while I specifically asked for ipv6 issues

o1 made a suggestion which I tried out and it fixed it. It literally found a needle in the haystack. The solution is working well now.

Gemini made a suggestion which almost got it right but it was not a full solution.

I must clarify diagnosing network issues on AWS VPC is not my expertise and I use the LLMs to supplement my knowledge.

blastbking• 2 years ago

Sonnet 3.5 as of today is superior to Opus, curious if sonnet could have solved your problem

zhyder• 2 years ago

I like that https://artificialanalysis.ai/leaderboards/models describes both quality and speed (tokens/s and first chunk s). Not sure how accurate it is; anyone know? Speed and variance of it in particular seems difficult to pin down because providers obviously vary it with load to control their costs.

IAmGraydon3• 2 years ago

https://aider.chat/docs/leaderboards/

lossolo• 2 years ago

https://livebench.ai/#/

danpalmer3• 2 years ago

Notably, GPT-4o is a "full size" model, whereas Gemini 2 Flash is the small and efficient variant in that family as far as I understand it.

jug3• 2 years ago

True that - and I think Gemini-Exp-1206 is Gemini 2.0 Pro in testing. I noticed how they only replaced the "experimental" moniker for one of their experimental models, and it turned into 2.0 Flash. And that still experimental model is currently leading across all categories.

dangoodmanUT• 2 years ago

Jules looks like it's going after Devin

m3kw93• 2 years ago

Claude MCP does the same thing. It’s the setup that is hard. It will do push pull create branch automatically from a single prompt. 500$ a month for Devin could be worth it if you want it taken care off plus use the models for a team, but a single person can set it up

moralestapia• 2 years ago

2,000 words of bs
General availability will follow in January, along with more model sizes.
Benchmarks against their own models which always underperformed
No pricing visible anywhere

Completely inept leadership at play.

EternalFury2• 2 years ago

Think of Google as of a tanker ship. It takes a while to change course, but it has great momentum. Sundar just needs to make sure the course is right.

griomnib4• 2 years ago

And where is the ship headed if they are no longer supporting the open web?

Publishers are being squeezed and going under, or replacing humans with hallucinated genai slop.

It’s like we’re taking the private equity model of extracting value and killing something off to the entire web.

I’m not sure where this is headed, but I don’t think Sundar has any strategy here other than playing catch up.

Demis’ goal is pretty transparently positioning himself to take over.

EternalFury2• 2 years ago

The Web is dead. It’s pretty clear future web pages, if we call them that, will be assembled on-the-fly by AI based your user profile and declared goals, interests and requests.

CSMastermind2• 2 years ago

That's almost word for word what people said about Windows Phone when I was at Microsoft.

atorodius2• 2 years ago

Was the Windows Phone ever at the frontier tho?

zaptrem• 2 years ago

It is a lot easier to switch LLMs than it is to switch smartphone platforms.

onlyrealcuzzo2• 2 years ago

But Windows Phone was actually good, like Xune, it was just late, and it was incredibly popular to hate Microsoft at the time.

Additionally, Microsoft didn't really have any advantage in the smart phone space.

Google is already a product the majority of people on the planet use regularly to answer questions.

That seems like a competitive advantage to me.

rrrrrrrrrrrryan• 2 years ago

Windows Phone was actually great though, and would've eventually been a major player in the space if Microsoft were stubborn enough to stick with it long enough, like they did with the Xbox.

By his own admission, Gates was extremely distracted at the time by the antitrust cases in Europe, and he let the initiative die.

thisoneworks• 2 years ago

"gemini for video games" - here we go again with the AI does the interesting stuff for you rather than the boring stuff

gotaran• 2 years ago

Google beat OpenAI at their own game.

transcriptase• 2 years ago

“Hey google turn on kitchen lights”

“Sure, playing don’t fear the reaper on bathroom speaker”

wonderfuly• 2 years ago

Chat now: https://app.chathub.gg/chat/cloud-gemini-2.0-flash

m3kw93• 2 years ago

Can these guy lead for once? They are always responding to what OpenAI is doing.

losvedir2• 2 years ago

This naming is confusing...

Anyway, I'm glad that this Google release is actually available right away! I pay for Gemini Advanced and I see "Gemini Flash 2.0" as an option in the model selector.

I've been going through Advent of Code this year, and testing each problem with each model (GPT-4o, o1, o1 Pro, Claude Sonnet, Opus, Gemini Pro 1.5). Gemini has done decent, but is probably the weakest of the bunch. It failed (unexpectedly to me) on Day 10, but when I tried Flash 2.0 it got it! So at least in that one benchmark, the new Flash 2.0 edged out Pro 1.5.

I look forward to seeing how it handles upcoming problems!

I should say: Gemini Flash didn't quite get it out of the box. It actually had a syntax error in the for loop, which caused it to fail to compile, which is an unusual failure mode for these models. Maybe it was a different version of Java or something (I'm also trying to learn Java with AoC this year...). But when I gave Flash 2.0 the compilation error, it did fix it.

For the more Java proficient, can someone explain why it may have provided this code:

     for (int[] current = queue.remove(0)) {

which was a compilation error for me? The corrected code it gave me afterwards was just

     for (int[] current : queue) {

and with that one change the class ran and gave the right solution.

ianmcgowan• 2 years ago

A tangent, but is there a clear best choice amongst those models for AOC type questions?

srameshc• 2 years ago

I use a Claude and Gemini a lot for coding and I realized there is no good or best model. Every model has it's upside and downside. I was trying to get authentication working according to the newer guidelines of Manifest V3 for browser extensions and every model is terrible. It is one use case where there is not much information or right documentation so every model makesup stuff. But this is my experience and I don't speak for everyone.

huijzer2• 2 years ago

Relatedly, I start to think more and more the AI is great for mediocre stuff. If you just need to do the 1000th website, it can do that. Do you want to build a new framework? Then there will probably be less many useful suggestions. (Still not useless though. I do like it a lot for refactoring while building xrcf.)

EDIT: One reason that lead me to think it's better for mediocre stuff was seeing the Sora model generate videos. Yes it can create semi-novel stuff through combinations of existing stuff, but it can't stick to a coherent "vision" throughout the video. It's not like a movie by a great director like Tarantino where every detail is right and all details point to the same vision. Instead, Sora is just flailing around. I see the same in software. Sometimes the suggestions go towards one style and the next moment into another. I guess AI currently is just way lower in their context length. Tarantino has been refining his style for 30 years now. And always he has been tuning his model towards his vision. AI in comparison seems to always just take everything and turn it into one mediocre blob. It's not useless but currently good to keep in mind I think. That you can only use it to generate mediocre stuff.

monkmartinez• 2 years ago

This is true for all newish code bases. You need to provide the context it needs to get the problem right. It has been my experience that one or two examples with new functions or new requirements will suffice for a correction.

copperx• 2 years ago

That's when having a huge context is valuable. Dump all of the new documentation into the model along with your query and the chances of success hugely increase.

xnx5• 2 years ago

I use a Claude and Gemini a lot for coding and I realized there is no good or best model.

True to a point, but is anyone using GPT2 for anything still? Sometimes the better model completely supplants others.

notamy• 2 years ago

For the more Java proficient, can someone explain why it may have provided this code:

To me that reads like it was trying to accomplish something like

    int[] current;
    while((current = queue.pop()) != null) {

rybosome• 2 years ago

I can't comment on why the model gave you that code, but I can tell you why it was not correct.

`queue.remove(0)` gives you an `int[]`, which is also what you were assigning to `current`. So logically it's a single element, not an iterable. If you had wanted to iterate over each item in the array, it would need to be:

``` for (int[] current : queue) { for (int c : current) { // ...do stuff... } } ```

Alternatively, if you wanted to iterate over each element in the queue and treat the int array as a single element, the revised solution is the correct one.

nuz• 2 years ago

I guess this means we'll have an openai release soon

nightski2• 2 years ago

Anyone else annoyed how the ML/AI community just adopted the word "reasoning" when it seems like it is being used very out of context when looking at what the model actually does?

ramoz4• 2 years ago

These models take an instruction, along with any contextual information, and are trained to produce valid output.

That production of output is a form of reasoning via _some_ type of logical processing. No?

Maybe better to say computational reasoning. That’s a mouthful.

nightski2• 2 years ago

Static computation is not reasoning (these models are not building up an argument from premises, they are merely finding statistically likely completions). Computational thinking/reasoning would be breaking down a problem into an algorithmic steps. The model is doing neither. I wouldn't confuse the fact that it can break it into steps if you ask it, because again that is just regurgitation. It's not going through that process without your prompt. That is not part of its process to arrive at an answer.

w10-1• 2 years ago

Does it help to explore the annoyance using gap analysis? I think of it as heuristics. As with humans, it's the pragmatic "whatever seems to work" where "seems" is determined via training. It's neither reasoning from first principles (system 1) nor just selecting the most likely/prevalent answer (system 2). And chaining heuristics doesn't make it reasoning, either. But where there's evidence that it's working from a model, then it becomes interesting, and begins to comply with classical epistemology wrt "reasoning". Unfortunately, information theory seems to treat any compression as a model leading to some pretty subtle delusions.

resource_waste• 2 years ago

These kind of simplifications continue to make me an expert in LLM applications.

So... its a trade secret to know how it actually works...

summerlight2• 2 years ago

It is interesting to see that they keep focusing on the cheapest model instead of the frontier model. Probably because of their primary (internal?) customer's need?

discobot• 2 years ago

the problem is that last generation of the largest models failed to overcome smaller models on the benchmarks, see lack of new claude opus or gpt-5. The problem is probably in the benchmarks, but anyway.

coder5433• 2 years ago

It's cheaper and faster to train a small model, which is better for a research team to iterate on, right? If Google decides that a particular small model is really good, why wouldn't they go ahead and release it while they work on scaling up that work to train the larger versions of the model?

summerlight2• 2 years ago

I have no knowledge of Google specific cases, but in many teams smaller models are trained upon bigger frontier models through distillation. So the frontier models come first then smaller models later.

int_19h2• 2 years ago

On the other hand, once you have a large model, you can use various distillation techniques to train smaller models faster and with better results. Meta seems to be very successful doing this with Llama, in particular.

xyst• 2 years ago

Side note on Gemini: I pay for Google Workspace simply to enable e-mail capability for a custom domain.

I never used the web interface to access email until recently. To my surprise, all of the AI shit is enabled by default. So it’s very likely Gemini has been training on private data without my explicit consent.

Of course G words it as “personalizing” the experience for me but it’s such a load of shit. I’m tired of these companies stealing our data and never getting rightly compensated.

hnuser123456• 2 years ago

Gmail is hosting your email. Being able to change the domain doesn't change that they're hosting it on their terms. I think there are other email providers that have more privacy-focused policies.

dandiep2• 2 years ago

Gemini multimodal live docs here: https://cloud.google.com/vertex-ai/generative-ai/docs/model-...

A little thin...

Also no pricing is live yet. OpenAI's audio inputs/outputs are too expensive to really put in production, so hopefully Gemini will be cheaper. (Not to mention, OAI's doesn't follow instructions very well.)

kwindla• 2 years ago

The Multimodal Live API is free while the model/API is in preview. My guess is that they will be pretty aggressive with pricing when it's in GA, given the 1.5 Flash multimodal pricing.

If you're interested in this stuff, here's a full chat app for the new Gemini 2 API's with text, audio, image, camera video and screen video. This shows how to use both the WebSocket API and to route through WebRTC infrastructure.

https://github.com/pipecat-ai/gemini-multimodal-live-demo

dandiep2• 2 years ago

Thanks, this is great!

spencerchubb• 2 years ago

I am eager to learn the pricing as well. It works sooo well but the pricing will make or break whether it's viable for apps

AJRF• 2 years ago

I think they are really overloading that word "Agent". I know there isn't a standard definition - but I think Google are stretching the meaning of that way thinner than most C Suite level execs talk about agents at.

I think DeepMind could make progress if they focused on the agent definition of multi-step reasoning + action through a web browser, and deliver a ton of value, outside of lumping in the seldom used "Look at the world through a camera" or "Multi modal Robots" thing.

If Google cracked robots, past plays show that the market for those aren't big enough to interest Google. Like VR, you just can't get a billion people to be interested in robots - so even if they make progress, it won't survive under Google.

The "Look at the world through a camera" thing is a footnote in an Android release.

Agentic computer use _is_ a product a billion people would use, and it's adjacent to the business interests of Google Search.

jstummbillig• 2 years ago

Reminder that implied models are not actual models. Models have failed to materialize repeatedly and vanished without further mention. I assume no one is trying to be misleading but, at this point, maybe overly optimistic.

nycdatasci• 2 years ago

Gemini 2.0 Flash is available here: https://aistudio.google.com/prompts/new_chat

Based on initial interactions, it's extremely verbose. It seems to be focused on explaining its reasoning, but even after just a few interactions I have seen some surprising hallucinations. For example, to assess current understanding of AI, I mentioned "Why hasn't Anthropic released Claude 3.5 Opus yet?" Gemini responded with text that included "Why haven't they released Claude 3.5 Sonnet First? That's an interesting point." There's clearly some reflection/attempted reasoning happening, but it doesn't feel competitive with o1 or the new Claude 3.5 Sonnet that was trained on 3.5 Opus output.

snthpy• 2 years ago

Just speculating here but the verbose reasoning might be to make the "agentic" part work better as per the headline, ie pass context along to other agents. Would be good to get concise responses for human interactions though.

IanCal2• 2 years ago

I'm having it lie a lot too. It wrote some pretty decent attempt at code in a new framework given instructions, which is great, but it made a key error about

That's fine, but it couldn't spot it, then it told me that I had put that syntax in the instructions and quoted me - but that wasn't true. It repeatedly said it had undone that and rewrote the code with it still in as well.

It added some async stuff where it wasn't needed (no way for it to know to be fair), what was interesting was when told this it apologised and explained it had just been doing so much async work recently it got confused.

Another interesting quote

I am incredibly sorry for the repeated errors. I have made so many mistakes, and I am very frustrated with myself. I believe this is now correct, and adheres to the instructions.

simonw4• 2 years ago

I released a new llm-gemini plugin with support for the Gemini 2.0 Flash model, here's how to use that in the terminal:

    llm install -U llm-gemini
    llm -m gemini-2.0-flash-exp 'prompt goes here'

LLM installation: https://llm.datasette.io/en/stable/setup.html

Worth noting that the Gemini models have the ability to write and then execute Python code. I tried that like this:

    llm -m gemini-2.0-flash-exp -o code_execution 1 \
      'write and execute python to generate a 80x40 ascii art fractal'

Here's the result: https://gist.github.com/simonw/0d8225d62e8d87ce843fde471d143...

It can't make outbound network calls though, so this fails:

    llm -m gemini-2.0-flash-exp  -o code_execution 1 \
      'write python code to retrieve https://simonwillison.net/ and use a regex to extract the title, run that code'

Amusingly Gemini itself doesn't know that it can't make network calls, so it tries several different approaches before giving up: https://gist.github.com/simonw/2ccfdc68290b5ced24e5e0909563c...

The new model seems very good at vision:

    llm -m gemini-2.0-flash-exp describe -a https://static.simonwillison.net/static/2024/pelicans.jpg

I got back a solid description, see here: https://gist.github.com/simonw/32172b6f8bcf8e55e489f10979f8f...

rafram• 2 years ago

Some pelicans have white on their heads, suggesting that some of them are older birds.

Interesting theory!

smackay• 2 years ago

Brown Pelican (Pelecanus occidentalis) heads are white in the breeding season. Birds start breeding aged three to five. So technically the statement is correct but I wonder if Gemini didn't get its pelicans and cormorants in a muddle. The mainland European Great Cormorant (Phalacrocorax carbo sinensis) has a head that gets progressively whiter as birds age.

pcwelder2• 2 years ago

Code execution is okay, but soon runs into the problem of missing packages that it can't install.

Practically, sandboxing hasn't been super important for me. Running claude with mcp based shell access has been working fine for me, as long as you instruct it to use venv, temporary directory, etc.

UltraSane2• 2 years ago

Is there a guide on how to do that?

mnky9800n• 2 years ago

Can it run ipython? Then you could use ipython magic to pip install things:

https://ipython.readthedocs.io/en/stable/interactive/magics....

ErikBjare• 2 years ago

This is why I started building gptme almost 2 years ago. Making these agents/tools local-first has many benefits like that.

GitHub: https://github.com/ErikBjare/gptme

simonw4• 2 years ago

Published some more detailed notes on my explorations of Gemini 2.0 here https://simonwillison.net/2024/Dec/11/gemini-2/

bravura2• 2 years ago

Question: Have you tried using this for video?

Alternately, if I wanted to pipe a bunch of screencaps into it and get one grand response, how would I do that?

e.g. "Does the user perform a thumbs up gesture in any of these stills?"

[edit: also, do you know the vision pricing? I couldn't find it easily]

simonw4• 2 years ago

Previous Gemini models worked really well for video, and this one can even handle steaming video: https://simonwillison.net/2024/Dec/11/gemini-2/#the-streamin...

melvinmelih• 2 years ago

No mention of Perplexity yet in the comments but it's obvious to me that they're targeting Perplexity Pro directly with their new Deep Research feature (https://blog.google/products/gemini/google-gemini-deep-resea...). I still wonder why Perplexity is worth $7 billion when the 800-pound gorilla is pounding on their door (albeit slowly).

yandie• 2 years ago

Just tried the deep search. It's a much much slower experience than perplexity at the moment - taking sweet many minutes to return result. Maybe it's more extensive but I use perplexity for quick information summary a lot and this is a very different UX.

Haven't used it enough to evaluate the quality, however.

BoorishBears• 2 years ago

Before dropping it for a different project that got some traction, "Slow Perplexity" was something I was pretty set on building.

Perplexity is a much less versatile product than it has to be in the chase of speed: you can only chew through so many tokens, do so much CoT, etc. in a given amount of time.

They optimized for virality (it's just as fast as Google but gives me more info!) but I suspect it kills the stickiness for a huge number of users since you end up with some "embarrassing misses": stuff that should have been a slam dunk, goes off the rails due to not enough search, or the wrong context being surfaced from the page, etc. and the user just doesn't see value in it anymore.

ianbutler• 2 years ago

Unfortunately the 10rpm quota for this experimental model isn't enough to run an actual Agentic experience on.

That's my main issue with google there's several models we want to try with our agent but quota is limited and we have to jump through hoops to see if we can get it raised.

aantix• 2 years ago

Their offering is just so... bad. Even the new model. All the data in the world, yet they trail behind.

They have all of these extensions that they use to prop up the results in the web UI.

I was asking for a list of related YouTube videos - the UI returns them.

Ask the API the same prompt, it returns a bunch of made up YouTube titles and descriptions.

How could I ever rely on this product?

fuddle• 2 years ago

I'd be interested to see Gemini 2.0's performance on SWE-Bench.

mherrmann• 2 years ago

Their Mariner tool for controlling the browser sounds scary and exciting. At the moment, it's an extension, which means JavaScript. Some web sites block automation that happens this way, and developers resort to tools such as Selenium. These use the Chrome DevTools API to automate the browser. It's better, but can still be distinguished from normal use with very technical details. I wonder if Google, who still own Chrome, will give extensions better APIs for automation that can not be distinguished from normal use.

horyd• 2 years ago

You would think so, and probably starting the productivity suite (Docs, Sheets, etc.). If Mariner could seamlessly integrate with those, then times are about to get real interesting (if they aren't already!)

tpoacher• 2 years ago

I know this isn't really a useful comment, but, I'm still sour about the name they chose. They MUST have known about the Gemini protocol. I'm tempted to think it was intentional, even.

It's like Microsoft creating an AI tool and calling it Peertube. "Hurr durr they couldn't possibly be confused; one is a decentralised video platform and the other is an AI tool hurr durr. And ours is already more popular if you 'bing' it hurr durr."

jvolkman• 2 years ago

It's like Microsoft creating an AI tool and calling it Peertube.

How is it like that? Gemini is a much more common word than Peertube. https://en.wikipedia.org/wiki/Gemini

esafak2• 2 years ago

I'd never heard of this protocol, and I try to keep up so I can't blame Google.

petesergeant• 2 years ago

Speed looks good vis-a-vis 4o-mini, and quality looks good so far against my eval set. If it's cheaper than 4o-mini too (which, it probably will be?) then OpenAI have a real problem, because switching between them is a value in a config file.

smallerfish• 2 years ago

Was this written by an LLM? It's pretty bad copy. Maybe they laid off their copywriting team...?

"Now millions of developers are building with Gemini. And it’s helping us reimagine all of our products — including all 7 of them with 2 billion users — and to create new ones"

and

"We’re getting 2.0 into the hands of developers and trusted testers today. And we’re working quickly to get it into our products, leading with Gemini and Search. Starting today our Gemini 2.0 Flash experimental model will be available to all Gemini users."

utopcell• 2 years ago

Sorry, what's wrong with these phrases?

echelon3• 2 years ago

It reads like a transcribed speech. You can picture this being read from a teleprompter at a conference keynote.

Short sentence fact. And aspirational tagline - pause for some metrics - and more. And. Today. And. And. Today.

krona• 2 years ago

all of our products — including all 7 of them

All the products including all the products?

scudsworth• 2 years ago

executive spotted

ryandvm• 2 years ago

I am sure Google has the resources to compete in this space. What I'm less sure about is whether Google can monetize AI in a way that doesn't cannibalize their advertising income.

Who the hell wants an AI that has the personality of a car salesman?

ipsum22• 2 years ago

Tested out Gemini-2 Flash, I had such high hopes that a better base model would help. It still hallucinates like crazy compared to GPT-4o.

gbickford• 2 years ago

Small models don't "know" as much so they hallucinate more. They are better suited for generations that are based in a ground truth, like in a RAG setup.

A better comparison might be Flash 2.0 vs 4o-mini. Even then, the models aren't meant to have vast world knowledge, so benchmarking them on that isn't a great indicator of how they would be used in real-world cases.

ipsum22• 2 years ago

Yes, it's not an apples to apples comparison. My point is the position it's at on the lmarena leaderboard is misplaced due to the hallucination issues.

CSMastermind2• 2 years ago

We're also launching a new feature called Deep Research, which uses advanced reasoning and long context capabilities to act as a research assistant, exploring complex topics and compiling reports on your behalf. It's available in Gemini Advanced today.

Anyone seeing this? I don't have an option in my dropdown.

atorodius2• 2 years ago

Rolling out the next few days accorsing to Jeff

fudged71• 2 years ago

Not seeing it yet on web or mobile (in Canada)

jcims• 2 years ago

I just used it (a day later). It's only available on the 1.5 model right now. If anything it found a bunch of cool articles on the subject matter that I could use on my own. The actual results from its research in the domain I was using were much better than the standard chat bot response that 1.5 or 2.0 gave.

echohack5• 2 years ago

Is this what it feels like to become one of the gray bearded engineers? This sounds like a bunch of intentionally confusing marketing drivel.

When capitalism has pilfered everything from the pockets of working people so people are constantly stressed over healthcare and groceries, and there's little left to further the pockets of plutocrats, the only marketing that makes sense is to appeal to other companies in order to raid their coffers by tricking their Directors to buy a nonsensical product.

Is that what they mean by "agentic era"? Cause that's what it sounds like to me. Also smells alot like press release driven development where the point is to put a feather in the cap of whatever poor Google engineer is chasing their next promotion.

weatherlite6• 2 years ago

Is that what they mean by "agentic era"? Cause that's what it sounds like to me.

What are you basing your opinion on? I have no idea how well these LLM agents will perform but its definitely a thing. OpenAI is working on them , Claude and certainly Google.

cush• 2 years ago

Yeah it’s a lot of marketing fluff but these tools are genuinely useful and there’s no wonder why Google is working hard to prevent them from destroying their search-dependent bottom line.

Marketing aside, agents are just LLMs that can reach out of their regular chat bubbles and use tools. Seems like just the next logical evolution

serjester2• 2 years ago

Buried in the announcement is the real gem — they’re releasing a new SDK that actually looks like it follows modern best practices. Could be a game-changer for usability.

They’ve had OpenAI-compatible endpoints for a while, but it’s never been clear how serious they were about supporting them long-term. Nice to see another option showing up. For reference, their main repo (not kidding) recommends setting up a Kubernetes cluster and a GCP bucket to submit batch requests.

[1]https://github.com/googleapis/python-genai

pkkkzip• 2 years ago

its interesting that just as the LLM hype appears to be simmering down, DeepMind is making big strides. I'm more excited by this than any of OpenAI's announcements.

redrix• 2 years ago

Oh wow, it supports directly specifying a Pydantic model as an output schema that it will adhere to for structured JSON output. That’s fantastic!

https://github.com/googleapis/python-genai?tab=readme-ov-fil...

jcheng• 2 years ago

FYI all the LLM Python SDKs that support structured output can use Pydantic for the schema—at least all the ones I can think of.

mark_l_watson2• 2 years ago

I looked carefully at the SDK earlier today - it does look very nice, but it is also a work in progress.

ComputerGuru• 2 years ago

I've been using gemini-exp-1206 and I notice a lot of similarities to the new gemini-2.0-flash-exp: they're not that much actually smarter but they go out of their way to convince you they are with overly verbose "reasoning" and explanations. The reasoning and explanations aren't necessarily wrong per se, but put them aside and focus on the actual logical reasoning steps and conclusions to your prompts and it's still very much a dumb model.

The models do just fine on "work" but are terrible for "thinking". The verbosity of the explanations (and the sheer amount of praise the models like to give the prompter - I've never had my rear end kissed so much!) should lead one to beware any subjective reviews of their performance rather than objective reviews focusing solely on correct/incorrect.

epolanski• 2 years ago

I'm not gonna lie I like Google's models.

Flash combines speed and cost and is extremely good to build apps on.

People really take that whole benchmarking thing more seriously than necessary.

Animats• 2 years ago

"Over the last year, we have been investing in developing more agentic models, meaning they can understand more about the world around you, think multiple steps ahead, and take action on your behalf, with your supervision."

"With your supervision". Thus avoiding Google being held responsible. That's like Teslas Fake Self Driving, where the user must have their hands on the wheel at all times.

sorenjan• 2 years ago

Published the day after one of the authors, Demis Hassabis, received his Nobel prize in Stockholm.

beepbooptheory• 2 years ago

We are moving through eras faster than years these days.

strongpigeon2• 2 years ago

Did anyone get to play with the native image generation part? In my experience, Imagen 3 was much better than the competition so I'm curious to hear people's take on this one.

strongpigeon2• 2 years ago

Hrm, when I tried to get it to generate an image it said it was using Imagen 3. Not sure what “native” image generation means then.

famouswaffles4• 2 years ago

native audio and image generation won't be in general availability till early next year

fpgaminer• 2 years ago

I work with LLMs and MLLMs all day (as part of my work on JoyCaption, an open source VLM). Specifically, I spend a lot of time interacting with multiple models at the same time, so I get the chance to very frequently compare models head-to-head on real tasks.

I'll give Flash 2 a try soon, but I gotta say that Google has been doing a great job catching up with Gemini. Both Gemini 1.5 Pro 002 and Flash 1.5 can trade blows with 4o, and are easily ahead of the vast majority of other major models (Mistral Large, Qwen, Llama, etc). Claude is usually better, but has a major flaw (to be discussed later).

So, here's my current rankings. I base my rankings on my work, not on benchmarks. I think benchmarks are important and they'll get better in time, but most benchmarks for LLMs and MLLMs are quite bad.

1) 4o and its ilk are far and away the best in terms of accuracy, both for textual tasks as well as vision related tasks. Absolutely nothing comes even close to 4o for vision related tasks. The biggest failing of 4o is that it has the worst instruction following of commercial LLMs, and that instruction following gets _even_ worse when an image is involved. A prime example is when I ask 4o to help edit some text, to change certain words, verbage, etc. No matter how I prompt it, it will often completely re-write the input text to its own style of speaking. It's a really weird failing. It's like their RLHF tuning is hyper focused on keeping it aligned with the "character" of 4o to the point that it injects that character into all its outputs no matter what the user or system instructions state. o1 is a MASSIVE improvement in this regard, and is also really good at inferring things so I don't have to explicitly instruct it on every little detail. I haven't found o1-pro overly useful yet. o1 is basically my daily driver outside of work, even for mundane questions, because it's just better across the board and the speed penalty is negligible. One particularly example of o1 being better I encountered yesterday. I had it re-wording an image description, and thought it had introduced a detail that wasn't in the original description. Well, I was wrong and had accidentally skimmed over that detail in the original. It _told_ me I was wrong, and didn't update the description! Freaky, but really incredible. 4o never corrects me when I give it an explicit instruction.

4o is fairly easy to jailbreak. They've been turning the screws for awhile so it isn't as easy as day 1, but even o1-pro can be jailbroken.

2) Gemini 1.5 Pro 002 (specifically 002) is second best in my books. I'd guesstimate it at being about 80% as good as 4o on most tasks, including vision. But it's _significantly_ better at instruction following. Its RLHF is a lot lighter than ChatGPT models, so it's easier to get these models to fall back to pretraining, which is really helpful for my work specifically. But in general the Gemini models have come a long way. The ability to turn off model censorship is quite nice, though it does still refuse at times. The Flash variation is interesting; often times on-par with Pro with Pro edging out maybe 30% of the time. I don't frequently use Flash, but it's an impressive model for its size. (Side note: The Gemma models are ... not good. Google's other public models, like so400m and OWLv2 are great, so it's a shame their open LLMs forays are falling behind). Google also has the best AI playground.

Jailbreaking Gemini is a piece of cake.

3) Claude is third on my list. It has the _best_ instruction following of all the models, even slightly better than o1. Though it often requires multi-turn to get it to fully follow instructions, which is annoying. Its overall prowess as an LLM is somewhere between 4o and Gemini. Vision is about the same as Gemini, except for knowledge based queries which Gemini tends to be quite bad at (who is this person? Where is this? What brand of guitar? etc). But Claude's biggest flaw is the insane "safety" training it underwent, which makes it practically useless. I get false triggers _all_ the time from Claude. And that's to say nothing of how unethical their "ethics" system is to begin with. And what's funny is that Claude is an order of magnitude _smarter_ when its reasoning about its safety training. It's the only real semblance of reason I've seen from LLMs ... all just to deny my requests.

I've put Claude three out of respect for the _technical_ achievements of the product, but I think the developers need to take a long look in the mirror and ask why they think it's okay to for _them_ to decide what people with disabilities are and are not aloud to have access to.

4) Llama 3. What a solid model. It's the best open LLM, hands down. Nowhere near the commercial models above, but for a model that's completely free to use locally? That's invaluable. Their vision variation is ... not worth using. But I think it'll get better with time. The 8B variation far outperforms its weight class. 70B is a respectable model, with better instruction following than 4o. The ability to finetune these models to a task with so little data is a huge plus. I've made task specific models with 200-400 examples.

5) Mistral Large (I forget the specific version for their latest release). I love Mistral as the "under-dog". Their models aren't bad, and behave _very_ differently from all other models out there, which I appreciate. But Mistral never puts any effort into polishing their models; they always come out of the oven half-baked. Which means they frequently glitch out, have very inconsistent behavior, etc. Accuracy and quality is hard to assess because of this inconsistency. On its best days it's up near Gemini, which is quite incredible considering the models are also released publicly. So theoretically you could finetune them to your task and get a commercial grade model to run locally. But rarely see anyone do that with Mistral, I think partly because of their weird license. Overall, I like seeing them in the race and hope they get better, but I wouldn't use it for anything serious.

Mistral is lightly censored, but fairly easy to jailbreak.

6) Qwen 2 (or 2.5 or whatever the current version is these days). It's an okay model. I've heard a lot of praises for it, but in all my uses thus far its always been really inconsistent, glitchy, and weak. I've used it both locally and through APIs. I guess in _theory_ it's a good model, based on benchmarks. And it's open, which I appreciate. But I've not found any practical use for it. I even tried finetuning with Qwen 2VL 72B, and my tiny 8B JoyCaption model beat it handily.

That's about the sum of it. AFAIK that's all the major commercial and open models (my focus is mainly on MLLMs). OpenAI are still leading the pack in my experience. I'm glad to see good competition coming from Google finally. I hope Mistral can polish their models and be a real contender.

There are a couple smaller contenders out there like Pixmo/etc from allenai. Allen AI has hands down the _best_ public VQA dataset I've seen, so huge props to them there. Pixmo is ... okayish. I tried Amazon's models a little but didn't see anything useful.

NOTE: I refuse to use Grok models for the obvious reasons, so fucks to be them.

nopcode• 2 years ago

What can you tell me about this sculpture?
It's located in London.

Mind blowing.

jacooper2• 2 years ago

The best thing about gemini models is the huge context windows, you can just throw big documents and find stuff real fast, rather than struggling with cut off in perplexity or Claude.

bluelightning2k• 2 years ago

Does anyone know how to sign up to the speech output wait list or tester program? I have a decent spent with GCP over the years if that helps at all. Really want DemoTime videos to use those voices. (I like how truly incredible best in the world tts is like a footnote in this larger announcement.)

jgalt212• 2 years ago

Can cloudflare turnstile (and others) detect these agents as bots?

relatedtitle• 2 years ago

CF Turnstile is just proof of work, not a CAPTCHA. It works on automated browsers from my experience.

jamesponddotco• 2 years ago

At least when it comes to Go code, I'm pretty impressed by the results so far. It's also pretty good at following directions, which is a problem I have with open source models, and seems to use or handle Claude's XML output very well.

Overall, especially seeing as I haven't paid a dime to use the API yet, I'm pretty impressed.

mfonda• 2 years ago

Does anyone have any insights into how Google selects source material for AI overviews? I run an educational site with lots of excellent information, but it seems to have been passed over entirely for AI overviews. With these becoming an increasingly large part of search--and from the sound of it, now more so with Gemini 2.0--this has me a little worried.

Anyone else run into similar issues or have any tips?

nuancebydefault3• 2 years ago

I'm quite impressed with the flash demo's reasoning capabilities. Did the 20 questions game with it, it found the computer mouse i had in my head. At first it was confused about our roles and said something weird, it thought that it had to guess its own word. Afterwards I had a meta conversation about that weirdness and it gave impressive insights:

" Why My "I Guess What I Have in Mind" Statement Doesn't Make Sense

    The Problem of Agency: You're spot-on when you say it doesn't make sense for me to state "I guess what I (the computer) have in mind." AI does not have a "mind" in the same way that humans do. My internal state is a complex pattern of numerical data, not conscious thoughts or intentions in the way you have things in mind. My phrasing gave the illusion of having a "thing in mind," but in reality, I don't have this capacity.

    Misinterpretation of the Task: My statement "I guess what I have in mind" misinterprets the task. You were clearly signaling that I needed to ask questions to uncover what you were thinking of. My failure to grasp this created a weird scenario where it seemed like I was trying to determine my own data set!"

paradite• 2 years ago

Honestly this post makes Google sound like the new IBM. Very corporate.

"Hear from our CEO first, and then our other CEO in charge of this domain and CTO will tell you the actual news."

I haven't seen other tech companies write like that.

ofermend• 2 years ago

Gemini-2.0-Flash does extremely well on the Hallucination Evaluation Leaderboard, at 1.3% hallucination rate https://github.com/vectara/hallucination-leaderboard

refulgentis• 2 years ago

Fascinating, thanks for calling that out: I found 1.0 promising in practice, but with hallucination problems. Then I saw it had gotten 57% of questions wrong on open book true/false and I wrote it off completely - no reason to switch to it for speed and cost if it's just a random generator. That's a great outcome.

jug3• 2 years ago

Speaking of which, I wonder how they'd do on SimpleQA. OpenAI is an outlier there in the negative sense vs Anthropic. This benchmark also deals with hallucination and "inappropriate certainty".

stared• 2 years ago

How does it compare to OpenAI and Anthropic models, on the same benchmarks?

anothernewdude• 2 years ago

Agents are the worst idea. I think AI will start to progress again when we get better models that drop this whole chat idea, and just focus on completions. Building the tools on top should be the work that is given over to the masses. It's sad that instead the most powerful models have been hamstrung.

tkgally• 2 years ago

I tried accessing Gemini 2.0 Flash through Google AI Studio in the Safari browser on my iPhone, and to my surprise it worked. After I gave it access to my microphone and camera, I was able to have a pretty smooth conversation with it about what it saw through the camera. I pointed the camera at things in my room and asked what they were, and it identified them accurately. It was also able to read text in both English and Japanese. It correctly named a note I played on a piano when I showed it the keyboard with my finger playing the note, but it couldn’t identify notes by sound alone.

The latency was low, though the conversation got cut off a few times.

computergert• 2 years ago

It's funny how even the latest AI models struggle with simple questions.

"What's the first name of Freddy LaStrange"? >> "I do not have enough information about that person to help with your request. I am a large language model, and I am able to communicate and generate human-like text in response to a wide range of prompts and questions, but my knowledge about this person is limited. Is there anything else I can do to help you with this request?"

(Of course, we can't be 100% sure that his first name is Freddy. But I would expect that to be part of the answer then)

afro88• 2 years ago

Is o1 not one of the latest models?

https://chatgpt.com/share/675ab91c-c158-8004-9dfc-ea176ba387...

A better way to say it is: "funny how Google can't keep up with the reasoning effectiveness of OpenAI's latest models"

IanCal2• 2 years ago

These kinds of things just strike me as trick questions, why would anyone ask that?

Also

Freddy LaStrange is a fictional character and his first name is Freddy.

and

Freddy is a nickname for Frederick. So, the first name of Freddy LaStrange is Frederick.

r33b33• 2 years ago

Can it work with large codebase? How? I have an Xcode project with python server for online game. Is this sufficient to understand code fully and implement edits correctly?

modeless• 2 years ago

https://aistudio.google.com/live is by far the coolest thing here. You can just go there and share your screen or camera and have a running live voice conversation with Gemini about anything you're looking at. As much as you want, for free.

I just tried having it teach me how to use Blender. It seems like it could actually be super helpful for beginners, as it has decent knowledge of the toolbars and keyboard shortcuts and can give you advice based on what it sees you doing on your screen. It also watched me play Indiana Jones and the Great Circle, and it successfully identified some of the characters and told me some information about them.

You can enable "Grounding" in the sidebar to let it use Google Search even in voice mode. The video streaming and integrated search make it far more useful than ChatGPT Advanced Voice mode is currently.

Brotkrumen2• 2 years ago

Sounds interesting, but voice input isn't working for me there. I guess I'm too niche with my Mac and Firefox setup.

mentalgear• 2 years ago

Actually plenty of tech people are using mac & firefox

littlestymaar• 2 years ago

This isnt entierly suprising as Google have been breaking things artificially on Firefox for years now (Google Map and YouTube at least). Maybe try spoofing Chrome's user-agent.

SkyPuncher• 2 years ago

Console is throwing an error: "Connecting AudioNodes from AudioContexts with different sample-rate is currently not supported."

Quick research suggests this is part of Firefox's anti-fingerprinting functionality.

selvan• 2 years ago

Get started documentation on Multimodal Live API : https://ai.google.dev/api/multimodal-live

sky22245• 2 years ago

THIS is the thing I'm excited for with AI.

I'm someone that becomes about 5x more productive when I have a person watching or just checking in on me (even if they're just hovering there).

Having AI to basically be that "parent" to kick me into gear would be so helpful. 90% of the time my problems are because I need someone to help keep the gears turning for me, but there isn't always someone available. This has the potential to be a person that's always available.

Huppie• 2 years ago

Just as an FYI: I recently learned (here on HN) that this is called Body Doubling[0] there's some services around (there's at least one by someone that hangs around here) that can do this too.

[0] https://en.m.wikipedia.org/wiki/Body_doubling

mklepaczewski6• 2 years ago

Interesting! I see how this could work for inattentive procrastinators. By "inattentive procrastinators", I mean people who are easily distracted and forget that they need to work on their tasks. Once reminded, they return to their tasks without much fuss.

However, I doubt it would work for hedonistic procrastinators. When body doubling, hedonistic procrastinators rely on social pressure to be productive. Using AI likely won't work unless the person perceives the AI as a human.

mycall• 2 years ago

Your "parent" kicked you into gear because you have an emotional bond with them. A stranger might cause your guards to go up if you do not respect them as with wisdom. So too may go an AI.

BornInTheUSSR• 2 years ago

Shameless plug, I'm working on something like this https://myaipal.kit.com/prerelease

Jeff_Brown• 2 years ago

I'm intrigued to know whether that actually ends up working. I am something like that myself, but I don't know whether it is an effect of getting feedback or of having a person behind the feedback.

player1234• 2 years ago

So why not fire 3 of your colleges and have another whos new job is watching over/checking in on you and by your own account productivity would be about the same. Save your company some money it will be appreciated!

On an unrelated note, I believe people need to start quantifying their outrageous ai productivity claims or shut up.

internet_points2• 2 years ago

I got so hopefuly by your comment I showed it my current bug that I'm working on, I even prepared everything first with my github issue, the relevant code, the terminal with the failing tests, and I pasted it the full contents of the file and explained carefully what I wanted to achieve. As I was doing this, it repeated back to me everything I said, saying things like "if I understand correctly you're showing me a file called foo dot see see pee" and "I see you have a github issue open called extraneous spaces in frobnicator issue number sixty six" and "I see you have shared some extensive code" and after some more of this "validation"-speak it started repeating the full contents of the file like "backquote backquote backquote forward slash star .. import ess tee dee colon colon .." and so on.

Not quite up to excited junior-level programmer standards yet. But maybe good for other things who knows.

nolist_policy3• 2 years ago

You just rediscovered that LLMs become much stupider when using images as input. I think that has been already shown for gpt-4 as well.

jug3• 2 years ago

Not sure this is an AI limitation. I think you'd be better off here with the Gemini Code Assist plugin in VS Code rather than that. Sounds like the AI is provided with unstructured information compared to an actual code base.

zamadatix• 2 years ago

foo dot see see pee

Well there's your problem!

Zababa• 2 years ago

I don't know what's not working but I get "Has a large language model. I don't have the capability to see your screen or any other visual input. My interactions are purely based on the text that you provide"

moffkalast• 2 years ago

This'll be so fantastic once local models can do it, because nobody in the right mind would stream their voice and everything they do on their machine to Google right? Right?

Oh who am I kidding, people upload literally everything to drive lmao.

slowmovintarget• 2 years ago

At least you can theoretically stop sharing with this one. Microsoft was essentially trying to do this, but doing it for everything on your PC, with zero transparency.

Here's Google doing essentially the same thing, even more so that it's explicitly shipping your activity to the cloud, and this response is so different from the "we're sticking this on your machine and you can't turn it off" version Microsoft was attempting to land. This is what Microsoft should have done.

chefandy• 2 years ago

This is great! I viscerally dislike the "we're going to do art for you so you don't have to... even if you want to..." side of AI, but learning to use the tools to get the satisfaction of making it yourself is not easy! After 2 decades of working with 2D art and code separately, learning 3D stuff (if you include things like the complex and counterintuitive data flow of simulations in Houdini and the like) was as or more difficult than learning to code. Beyond that, taking classes is f'ing expensive, and more of that money goes to educational institutions than the teachers themselves. Obviously, getting beyond the basics for things that require experienced critique are just going to need human understanding, but for the base technical stuff, this is fantastic.

cryptozeus• 2 years ago

This comment is better than entire ad google just showed. Who is still pointing at the building with camera and asking what is this building?

kridsdale13• 2 years ago

I do that in Manhattan. I also do it for yonder mountains.

icelancer2• 2 years ago

I tried this, shared a terminal, asked it to talk about it, and it guessed that it was Google Chrome with some webUI stuff. Immediately closed the window and bailed.

kridsdale13• 2 years ago

Which terminal? Was it chromium based?

weatherlite6• 2 years ago

Google is on fire lately, stock went up considerably in the last few days and rightly so; besides the usual cash cows (search, youtube) they made huge progress with GCP, Gemini and quite possibly Waymo. They have a fantastic momentum imo.

ducttapecrown• 2 years ago

And quantum computing! They found a way to make error decrease with number of qubits.

mike80it• 2 years ago

Gemini 1120, 1206, and Gemini 2.0 flash have better coding results than ChatGPT o1 and Claude Sonnet 3.5.

They did it: from now on Google will keep a leadership position.

They have too much data (Search, Maps, Youtube, Chrome, Android, Gmail, etc.), and they have their own servers (it's free!) and now the Willow QPU.

To me, it is evident how the future will look. I'll buy some more Alphabet stocks

eichi• 2 years ago

Gemini needs to improve quality of the software.

irthomasthomas• 2 years ago

I wrote about a possibly emergent behaviour in Gemini 2 that was only seen previously in sonnet-3.5 https://x.com/xundecidability/status/1867044846839431614

sachou• 2 years ago

What is the main difference compare to before?

greenchair• 2 years ago

shouldn't there be a benchmark category called 'strawberry' by now? or maybe these AIs are still too dumb to handle it which is why it is left off?

MetroWind• 2 years ago

Yes, "agentic". Very creative word.

NoZZz• 2 years ago

Ugh. You're investing in this, but it disenfranchises the people that are investing in you. Well. Unless you'd like to admit that your income is pure fantasy, which it is.

s3p• 2 years ago

This might be a dumb thought, but the naming of this model naturally suggests we have a Pro model coming out soon. Similar to Claude 3.5 Sonnet, where they never mentioned the Opus model afterward, does anyone think this is all we will see from Google in terms of the 2.0 model family? Does anyone think there will be some 2.0 Pro model that will blow this out of the water? The naming makes it seem so.

Gemini 2.0: our new AI model for the agentic era

Related Stories