Google will let companies run Gemini models in their own data centers

https://www.cnbc.com/2025/04/09/google-will-let-companies-run-gemini-models-in-their-own-data-centers.html

by jonbaer • 1 year ago

509 182 1 year ago

rwmj• 1 year ago

A bit thin on detail, but will this require confidential VMs with encrypted GPUs? (And I wonder how long before someone cracks SEV-SNP and TDX and pirate copies escape into the wild.)

unsnap_biceps• 1 year ago

The number of folks that have the hardware at home to run it is going to be very low and the risk of companies for leaking it is gonna make it unlikely IMHO.

notpushkin• 1 year ago

I think home users would be the least of their concerns.

RadiozRadioz• 1 year ago

It only takes one company to leak it

BiteCode_dev• 1 year ago

They can get "hacked" and wooops.

vasco• 1 year ago

At the pace models improve, the advantage of going the dark route shouldn't really hold for long, unless I'm missing something.

miohtama2• 1 year ago

Access to proprietary training data: Search, YouTube, Google Books might give some moat.

bjackman• 1 year ago

I wonder how long before someone cracks SEV-SNP

https://bughunters.google.com/blog/5424842357473280/zen-and-...

NoahZuniga• 1 year ago

I'd expect watermarked model weights plus a lot of liability to distinctivise leaking the model.

blitzar• 1 year ago

Is The Gavin Belson Signature Edition Box is needed to run these?

unixhero• 1 year ago

No, Jack Barkers' revamped version is needed.

FirmwareBurner2• 1 year ago

I miss that show. Too bad it ended right before the AI hype.

radicalbyte• 1 year ago

Fingers crossed that it'll do an Arrested Development.

fonsai• 1 year ago

Wasn't the last season mainly about AI?

j_bum• 1 year ago

I miss it too.

I think by the end I was far more invested in the characters rather than the plot though.

zer00eyz• 1 year ago

Too bad it ended right before the AI hype.

It was the very start of the AI hype cycle, and in fact they built the app: https://news.ycombinator.com/item?id=14636228

swalsh• 1 year ago

The google search appliance might have been one of the worst products I've ever used in my career. If they're going to make a box, I hope they put some effort into it.

mosura• 1 year ago

Strictly speaking Google still make boxes for people just in a different market.

What was so bad about the search appliance though? Physical? Software?

stingraycharles2• 1 year ago

The Netflix appliance is pretty good in my experience. No reason Google couldn’t pull something similar off themselves, unless they’re being very Google about it.

er4hn• 1 year ago

I actually mourn the loss of it. It felt so much better than any other accursed on-prem search solution I've seen since.

dehrmann2• 1 year ago

My theory is that the heuristics (PageRank and click-through feedback) that made 2008 Google great don't work in corporate environments.

runjake• 1 year ago

Why? In my limited experience, it was pretty useful, but again, my experience is limited.

What made it one of the worst products you’ve ever used?

sorokod• 1 year ago

More like the GSA probably.

https://en.m.wikipedia.org/wiki/Google_Search_Appliance

larodi• 1 year ago

which is a black-box more or less, so the "own data center" is a bit vague as concept here.

surajrmal8• 1 year ago

You're going to need a few specialized racks all wired up together. A single box won't be sufficient.

ashoeafoot• 1 year ago

He personally stands behind the developed haedware, will you stand with him ?

next_xibalba• 1 year ago

This tactic comes straight out of the Conjoined Triangles of Success playbook. It’s a classic Action Jack Barker move.

hinkley3• 1 year ago

On paper, Stephen Tobolowsky seems like he shouldn’t be successful enough of an actor to warrant an autobiography. But man do I love Ned “The Head” Ryerson in all his incarnations. What a strange, tall, little man.

That the world does not have a Stanley Tucci, Stephen Tobolowsky buddy comedy trilogy has made it all the poorer. But it’s been a while since someone tried to remake The Odd Couple…

nkassis2• 1 year ago

His signature could be the size of a full size rack on this one.

atum47• 1 year ago

I got that reference

hank808• 1 year ago

FTA: As part of the announcement, Google said Nvidia will bring Gemini models to the company’s Blackwell graphics processing units, or GPUs. Companies can buy the chips through Google or other channels.

fxtentacle2• 1 year ago

I believe these are pure word tricks to suggest privacy without actually delivering it.

As context, you need to remember that Google deleted their "Don't Be Evil" motto and became a defense contractor. The customer will most likely receive a black box owned and set up by Google. That means they have no way of knowing if the system inside is phoning home or being remote controlled by an US government agency, or not. You can then say that the model is hosted in your own data center, which might make some people feel good, but using it with personal information is still a violation of the GDPR.

If Google, however, would make these boxes fully offline capable and I was also allowed to wipe all hard disks myself before returning it, that would convince me of their good intentions.

greggsy2• 1 year ago

You’re making a lot of assumptions there. It’s trivial to monitor traffic patterns from modern appliances, even if it’s encrypted.

Also, companies have been sharing data with cloud security organisations for years now. There a robust means of assessing the risk. License agreements are a very real thing.

fxtentacle2• 1 year ago

I don't fully disagree, but the only reason why this product is noteworthy is precisely because companies don't trust cloud providers with their data anymore. And while you might be able to prevent data exfiltration by monitoring the traffic patterns, you probably can't prevent sabotage that way.

fhd2• 1 year ago

Well, TFA appears to be thin on the details, but who says whatever they deploy is phoning home? If you run their model on prem, it wouldn't be a difficult feat to monitor its network traffic. Not to mention limiting it. It would be tricky if it phoned home by design, but if this is all abstracted through tool use or something, it can certainly be audited. And the kind of company that wants this usually doesn't just run random software without understanding and inspecting closely what it does.

positr0n• 1 year ago

You’re talking about Fortune 50 companies here. I don't think Google is going to be messing around spying on them in direct violation of the no-doubt sophisticated contract that will be signed between them.

threeducks2• 1 year ago

The NSA has been spying through Google since at least 2009. https://en.wikipedia.org/wiki/PRISM

mkl2• 1 year ago

The boxes can run airgapped: https://cloud.google.com/blog/products/ai-machine-learning/r...

surajrmal8• 1 year ago

Why is don't be evil relevant here? If Google never had that motto would you care less? It's not even factual that they dropped it from the code of conduct. It was just moved to the end rather than at the beginning. Moving it wasn't some magical event that signaled a change in Google's ethical values. Do the right thing was just seen as less ambiguous and placed more prominently.

As others have stated, being able to see that the appliance is phoning home or not is trivial. No one who is in the market for this won't ensure it meets some rigurous bar.

cavisne3• 1 year ago

This is being sold as an air gapped product, it has to work offline by definition.

Sure you could hide some way of phoning home and deploy it into the SCIF, but would you really want to risk a firing squad to improve some advertising metrics?

holografix• 1 year ago

This is obvious government contract baiting. Kudos though they might actually move some Google Distributed Cloud this way

aduffy2• 1 year ago

I don’t think so. To my knowledge GCP has no approval for classified networks, which is by far the hardest part. Contrast with Azure OpenAI has been approved to run on government networks for over a year now.

This feels like a play for companies in highly regulated industries, GCP has a notable list of biopharma customers.

ZeroCool2u• 1 year ago

FedRAMP High is the mark you really want to hit for the US Government and GCP's service coverage is surprisingly broad in that realm.

skybrian• 1 year ago

From Google's blog post:

Our GDC air-gapped product, which is now authorized for US Government Secret and Top Secret missions, and on which Gemini is available, provides the highest levels of security and compliance.

Maxious• 1 year ago

Today at Google Cloud Next, we're thrilled to announce another significant milestone for Google Public Sector: the authorization of Google Distributed Cloud Hosted (GDC Hosted) to host Top Secret and Secret missions for the U.S. Intelligence Community, and Top Secret missions for the Department of Defense (DoD).

https://cloud.google.com/blog/topics/public-sector/google-pu...

nkassis2• 1 year ago

Banking as well, this is the kind of offering they've been looking for a while. Google just saw the demand decided to jump in while OpenAI and Anthropic probably calculated they don't have the manpower to deal with the support for this.

reaperducer3• 1 year ago

This is obvious government contract baiting

You don't have to be a government agency to not want your company's data all over the place.

connicpu• 1 year ago

With a few exceptions for companies with highly secretive data, you do have to be a government agency or working in a highly regulated government-adjacent area for secured private clouds to be a requirement carved in stone and therefore worth investing a ton of extra money into though.

culopatin• 1 year ago

They’ll have to fight Microsoft who’s been promising copilot.

noitpmeder• 1 year ago

Financial firms with significant on-prem datacenter use will love this as well. My company still stays away from the cloud -- we have 6 DCs in the building, and run everything else out of colocated racks.

brcmthrowaway• 1 year ago

Who provides internet

miohtama2• 1 year ago

Is Gemini tied/benefitting from Google TPU hardware? Because you need hardware in the data center to run this, and I feel it is somewhat specialised.

disgruntledphd2• 1 year ago

The Google blogpost notes that it's a partnership with Nvidia, so using cuda rather than TPUs apparently.

stingraycharles2• 1 year ago

Makes complete sense, as NVidia has a lot more experience building these types of appliances.

MortyWaves• 1 year ago

Google abandoned Coral in true Google style.

drdirk• 1 year ago

Gemini models are written in Jax which through the XLA compiler can be compiled either to TPU or GPU hardware.

Performance may differ but Google (and Nvidia) are very interested in having good performance on both platforms.

cavisne3• 1 year ago

The raw computation is just a bunch of matrix multiplications in a row, most of the algorithmic complexity/secret stuff would be around scaling & efficiency.

For training the model the HW is much more important as you need to scale up to as many chips as possible without being bottlenecked by the network.

This would just be inference, and it doesn't need to be very efficient as its for on prem usage not selling API access. So you could strip out any efficiency secrets, and it would probably look like a bigger Gemma (their open source model).

I wonder if they would/could try and strip out stuff like whatever tricks they use for long context + video support (both of which they are a bit ahead of everyone else on).

summerlight• 1 year ago

The model itself is likely built upon their own open source system JAX so they should be usable in Nvidia. Of course cost efficiency is going to be a different story.

Workaccount22• 1 year ago

TPUs are definitely the reason why Gemini models have both massive context and very low prices. There is no nvidia tax to pay.

amelius• 1 year ago

It's still an advertising company you're doing business with.

I mean, would you buy cookies from a brand that is known for producing rodenticides?

crazygringo3• 1 year ago

...yes?

I mean, the company that makes Raid also makes Saran Wrap and Ziploc bags. Corporate conglomerates can do lots of things.

The entire Google Cloud org is funded by regular customers paying money, not advertising.

righthand5• 1 year ago

The entire Google Cloud org is funded by regular customers paying money

That can’t be true, how did they bootstrap it? How do they pay for R & D for their half baked offerings?

NicuCalcea• 1 year ago

I buy Lidl store brand biscuits as well as Lidl store brand cleaning products, among many other things, so I guess I would.

rafaelmn• 1 year ago

Lidl produces none of those, just brands them. More comparable would be something like Raid cookies I guess

paxys2• 1 year ago

Google Cloud had $50B in revenue last year, so clearly plenty of companies didn't get your memo.

aussieguy1234• 1 year ago

What is the risk that some hacker could exfliltrate the weights?

dist-epoch• 1 year ago

Very low if they use confidential VMs (CPU rooted encryption). Just like the Xbox uses and remains unhacked 10 years later.

onlyrealcuzzo• 1 year ago

What would be the benefit of hacking the Xbox? What would you get?

cavisne3• 1 year ago

Seems pretty high, this is an air gapped product so at some point the employees of whatever government they are giving it to would need to SSH into the VM's to load new weights etc. Lots of ways to make it tricky/watermark the weights though.

djoldman• 1 year ago

Google announcement:

https://cloud.google.com/blog/products/ai-machine-learning/r...

mkl2• 1 year ago

Interesting that the hardware is NVidia Blackwell, not Google TPUs. That means Google will likely have an energy efficiency and cost advantage, and keep their proprietary hardware out of other people's reach.

bitexploder2• 1 year ago

Part of the reason for this is likely due to customers preference to have CUDA available which TPUs do not support. TPU is superior for many use cases but customers like the portability of targeting CUDA

re-thc• 1 year ago

not Google TPUs

They're in limited supply. Even Google doesn't have enough for their own use.

crowcroft• 1 year ago

Getting a whole business set up to build TPU hardware for third parties (design, build, sell, support, etc.) is probably not worth it when there is overflowing demand for TPUs in their cloud already.

Businesses running their own hardware probably prefer CUDA as well for being more generally useful.

WalterGR• 1 year ago

Google doesn’t make TPUs available to 3rd parties, right? I assume there would be tremendous reverse-engineering risk if they were to include them?

DSingularity• 1 year ago

It’s telling that their effort was to get an air gapped solution cleared for US government and the US military.

throwaway484762• 1 year ago

Like you can with deep seek? Or will it be more complicated and expensive. I don't know who would actually want that.

yoavm2• 1 year ago

Absolutely many would, especially those with deep pockets. The biggest concern I'm hearing from companies adopting AI, for basically any use case, is data leaving their network. Especially (but not only) in the EU.

throwaway484762• 1 year ago

Deepseek is just the model weights. Nothing about it requires network access.

surajrmal8• 1 year ago

Folks who would prefer to run deepseek are not in the end customer for this product. Deepseek doesn't provide a service contract.

nsriv• 1 year ago

This might be a great way for them to strengthen their model through federated learning.

https://federated.withgoogle.com/

ein0p4• 1 year ago

The whole point of deploying such things on-prem is air-gapping it from Google and its "learning".

fc417fc8023• 1 year ago

That's the point of the privacy scheme. It would only be able to learn things common to multiple clients. Private data wouldn't make it through the noise.

amarcheschi• 1 year ago

I did my undergrad internship on federated learning. I was tasked with implementing in a simulator different federated algorithms, so to have a way to compare them in a meaningful way. The last that had to be implemented was FedMA. We didn't manage to do it. That algorithm is absolutely devilish. Every issue that I solved made other two issue arise, and neither my supervisors could help. The sheer idea of matching neurons in different networks might (and does) make sense, but the way the approximate costs are calculated require other 2/3 math papers that I could follow for only the first lines of the abstract. I'm happy for the time I spent in my internship there. I'm also happy it's over

The general understanding of how it works is surprisingly easy though, you can find the paper here https://arxiv.org/abs/2002.06440

jaggs• 1 year ago

Not a cat's chance in hell that any eu organisation will rush to this offer right now. Or maybe ever in fact.

geodel• 1 year ago

Huh, how does that matter? Maybe OVH AI is fine for EU.

tziki• 1 year ago

I don't understand how Google is willing to do this but won't sell TPUs to other days centers. It should be obvious from Nvidia's market cap that they're missing a huge opportunity.

dehrmann2• 1 year ago

The only reasons I can think of is they see them as their secret sauce, they don't want to support them for customers long-term, or they don't have the foundry capacity.

paxys2• 1 year ago

It's definitely #3. The GPUs have to first satisfy Google's own computing needs, and only then can they start selling them to others. Given how much training and inference the company is doing and how much demand there is internally it's very unlikely they are able to manufacture loads of extras, especially not profitably.

_cs2017_• 1 year ago

Curious if this was forced on Google Cloud by Sundar, or was it something that Google Cloud as an org wanted to do?

At first glance, it seems Google Cloud might lose some revenue from customers who can now deploy Gemini in-house. On the other hand, it's not a complete loss, since presumably Google Cloud is still involved in providing some underlying tech? Not to mention, some customers would never consider using off-premises setup anyway.

wmf• 1 year ago

I assume Google Distributed Cloud is part of the larger Cloud org so they get the revenue either way. The on-prem version may even cost more.

ein0p4• 1 year ago

What a sudden change of heart. Thank you, DeepSeek!

bushbaba• 1 year ago

Reminds me of the Google search appliance.

jonhohle• 1 year ago

That’s the first thing I thought of as well. I had to integrate one into our custom CMS early in my career. I vaguely remember explaining to management that I was not responsible for the order or quality of search results and tweaking queries (now prompts?) with hints to restrict searches to certain paths. It was such an opaque device, but provided better results than MySQL did at the time.

rr808• 1 year ago

I work for a bank and we're banned from using AI because of the privacy problem. I'd think there is a big market for a product like this.

replete• 1 year ago

Google AI rackmount appliance coming soon?

Google will let companies run Gemini models in their own data centers

Related Stories