The Google employees who created transformers

https://www.wired.com/story/eight-google-employees-invented-modern-ai-transformers-paper/

by thm • 2 years ago

415 226 2 years ago

CharlesW2• 2 years ago

whatever1• 2 years ago

I am all in for allowing tax free bloated RnD departments just with the hope that once in a decade idea will propell the overall economy.

The marvel of modern computing is a result of RnD bloat that was done without immediate impact to the bottomlines of their own companies.

utopcell• 2 years ago

Tax-free ? This was a regular Google team.

bozhark• 2 years ago

Maybe they meant revenue free?

orev• 2 years ago

In the US, R&D expenses can often be deducted when filing taxes. I’m sure large companies like Google would be taking advantage of those rules.

xhkkffbf• 2 years ago

What they meant was that the R&D was fully deductible, not amortized over a number of years. It's like saying that a business's electricity bill is "tax-free" because they can deduct it from their revenues immediately.

And as it is, Google certainly paid property taxes immediately on the office building as well as FICA on all of the employees who, of course, paid their own taxes.

But haters of the R&D system love to call it "tax-free".

abofh• 2 years ago

Many inventions of the modern age were largely R&D bloat as you'd describe it - in this age R&D is done in university, spun out into a startup, and dies shortly after the IPO - but there's a flicker of it occasionally lasting on if it can hide under the guise of a tax deduction.

TremendousJudge• 2 years ago

Weren't transistors specifically invented to improve telephone networks?

amanda99• 2 years ago

I mean this is how the US operates. Other countries continually try to refine their existing industries and societal structures. In the US, you let everything rot and hope that a hail Mary innovation comes out of somewhere and creates a breath of fresh air. It creates quite a wild society, but it seems to have worked for now!

fennecbutt• 2 years ago

That would make sense, if companies and executives paid their damn taxes in the first place.

saadatq• 2 years ago

Of note:

“ Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively.”

IncreasePosts2• 2 years ago

Why would that be of note, especially in America? I think it would be an interesting observation in China or Japan, or some other country which is generally less welcoming to immigrants than the US

svnt4• 2 years ago

First generation immigrants are still a tiny minority of the population. The fact that the entire team consists effectively of first generation immigrants says something, probably both about higher education and American culture.

quantum_state• 2 years ago

The US is still the place to be, despite its problems…

AnimalMuppet2• 2 years ago

Because there are plenty of people in the US who are neither immigrants nor the children of immigrants. In fact, they're probably a significant majority. So to have 8 out of 8 be members of the smaller set is rather unlikely.

pksebben2• 2 years ago

less welcoming to immigrants than the US

Not sure if we can claim this any more, what with texas shipping busloads of immigrants to NY and the mayor declaring it a citywide emergency, and both major parties rushing to get a border wall built.

pyb• 2 years ago

With the current state of Visas for the US, most countries are now more welcoming to migrants.

bugglebeetle2• 2 years ago

Yes, this is one of the actually admirable qualities of the US and California in particular. CA has one of the world’s largest economies because it attracts and embraces people from just about every part of the world.

laidoffamazon• 2 years ago

The more interesting thing to me is that only one of them went to an elite American undergrad (Duke). Rest went to undergrads in India, Ukraine, Germany, Canada (UToronto has a 43% acceptance rate).

whywhywhywhy• 2 years ago

Looking forward to the inevitable movie about how they invented it and the company failed to productize it because of it's culture and management

CharlesW2• 2 years ago

Xerox 2: Electric Boogoogle

galoisscobi• 2 years ago

Same thing is happening with the Figure One robot. There isn't much new in the demo Figure did recently, all the work has been published in the palm saycan paper and other deepmind papers. Google is good at laying golden eggs and not realizing that they've laid them. Then employees get fed up, leave and start their own companies so they can productize their work without management getting in the way.

UberFly• 2 years ago

This is a story that could be written over and over again. Companies with big R&D budgets but not nimble enough any more to take advantage.

photochemsyn• 2 years ago

"Realistically, we could have had GPT-3 or even 3.5 probably in 2019, maybe 2020. The big question isn’t, did they see it? The question is, why didn’t we do anything with the fact that we had seen it? The answer is tricky.”

The answer is that monopolies stifle technological innovation because one well-established part of their business (advertising-centric search) would be negatively impacted by an upstart branch (chatbots) that would cut into search ad revenue.

This is comparable to a investor-owned consortium of electric utilities, gas-fired power plants, and fracked natural gas producers. Would they want the electric utility component to install thousands of solar panels and cut off the revenue from natural gas sales to the utility? Of course not.

It's a good argument for giving Alphabet the Ma Bell anti-trust treatment, certainly.

tyree731• 2 years ago

On the other hand, Alphabet's inability to deploy GPT-3 or GPT-3.5 has led to the possibility of its disruption, so anti-trust treatment may not be necessary.

IX-103• 2 years ago

Disrupted by a whom? Microsoft? Facebook? The company formerly known as Twitter? Even if one of these takes over we'd just be trading masters.

And that's ignoring how Alphabet's core business, Search, has little to fear from GPT-3 or GPT-3.5. These models are decent for a chatbot, but for anything where you want reliably correct answers they are lacking.

lelag6• 2 years ago

A better example of that behaviour would be Kodak which invented the first digital camera as soon as 1975 then killed the project, because it was a threat to their chemical business.

jl62• 2 years ago

Digital photography works not just because of the camera but because of the surrounding digital ecosystem. What would people do with digital photos in 1975?

tsunamifury3• 2 years ago

I was privy to the details of this. Sundar blocked the NEMA team from presenting at I/O with its gpt3 level product in the form of a character.

Sundar was afraid of the technology and how it would be received and tried to ice it.

jimbob45• 2 years ago

How is that not immediate grounds for his termination? The board should be canned too for allowing such an obvious charlatan to continue ruining the company.

skadamat• 2 years ago

Honestly, this is part of the reason why I don't think Google will be a dominant business in 10 years. Searching the web for information helped us service a lot of useful 'jobs to be done' ... but most of those are now all done better by ChatGPT, Claude, etc.

Sure we have Gemini but can Google take a loss in revenue in search advertising in their existing product to maybe one day make money from Gemini search? Advertising in the LLM interface hasn't been figured out yet.

Google (kind of) feels like an old school newspaper in the age of the internet. Advertising models for the web took a while to shake out.

bigcat123456786• 2 years ago

Transformer == Modern AI? Wow, I dont know where should I start My mind literally stuck at the absurdity

ozymandias1337• 2 years ago

AI is in the eye of the beholder.

supportengineer• 2 years ago

And in this case, the transformers are more than meets the eye.

itsoktocry• 2 years ago

It's 2024; do we really have to point out the fact that publications take liberties with headlines? It's been that way forever.

Besides that, are you denying that transformers are the fundamental piece behind all the AI hype today? That's the point of the article. I think the fact that a mainstream publication is writing about the Transformers paper is awesome.

YeGoblynQueenne• 2 years ago

OK, so the title should be changed to "8 Google Employees Invented Modern AI Hype"?

tsunamifury3• 2 years ago

AI hipsterism

bigcat123456786• 2 years ago

Modern AI is such a big term that its way off the scale of normal click-baiting...

Modern AI should anything after https://en.wikipedia.org/wiki/Dartmouth_workshop Not transformer

AI has been people's dream since written history. i.e., everyone in their own sense would want to invent something that can do things and think for themselves. That's literally the meaning of AI.

calf2• 2 years ago

What makes transformers the fundamental piece over say, big datasets, or Moore's law, or deep nets in general, etc?

J_Shelby_J• 2 years ago

Do you think this current AI hype machine came from something other than transformers?

golol3• 2 years ago

No, but it is weird to use "modern" here. Modern suggest a longer timeframe. I would say machine learning deep NNs is modern AI. It's just not true that everything not transformer is outdated, but it is "kinda" true that everything not deep NN is outdated.

mindcrime2• 2 years ago

No, but "modern AI" and "this current AI hype machine" are not equivalent phrases.

bigcat123456786• 2 years ago

Current AI hype comes from GPT

Transformer+Scaling+$$$ triggered the current hype

startupsfail• 2 years ago

Transformer is an evolution of large scale training of seq2seq. Diffusion models is an evolution of even older approach - training denoising autoencoders.

It had been a relatively gradual and accelerating progress, with number of people working in the field increasing exponentially, since 2010 or so, when Deep Learning on GPUs was popularized at NIPS by Theano.

Tens of thousands people working together on Deep Learning. Many more on GPUs.

serjester• 2 years ago

Modern AI != Machine Learning. Artificial intelligence has always been about models being able to accomplish a broad swath of tasks that they weren't explicitly trained on. It seems hard to argue we had accomplished this before massive transformers came along.

mkl95• 2 years ago

I thought level 4+ self-driving was going to be the face of "modern AI"... instead we got this thing

lelag6• 2 years ago

Why do you think transformers won't be key in making a level 4+ self-driving AI. It seems to me that Vision-capable multi-modal transformers could be the missing part: they can understand what is happening in the world in a deductive way.

The Vision transformer is capable of predicting that a running child is likely to follow that rolling soccer ball. It is capable of deducting that a particular situation looks dangerous or unusual and it should slow down, or stay away from danger in ways that previous crop of AI could not.

Imo, the only thing currently preventing transformers to change everything is the large amount of compute power required to run them. It's not currently possible to imagine GPT4-V running on a embedded computer inside a car. Maybe AI asic type of chip will solve that issue, maybe Edge computing and 5G will find it's use-case... Let's wait and see, but I would bet that transformer will find it's way in many places and change the world in many more ways than bringing us chatbots.

KerrAvon• 2 years ago

Bandwidth alone isn’t what prevents 5G from this sort of application, at least in the USA. Coverage maps tell the story: coverage is generally spotty away from major roads. Cell coverage isn’t a fixable problem in the near term, because every solution for doing that intersects with negative political externalities (antivax, NIMBYism, etc); if you can get people vaccinated for measles consistently again, then we can talk.

It all needs to be onboard. That’s where money should be going.

TimPC2• 2 years ago

I think we've found repeatedly in self-driving that it's not enough to solve the problem in the normal case. You need an AI model that has good behaviors in the edge cases. For the most part it won't matter how good the vision models get, you're going to need similar models that can make the same predictions from LIDAR signals because the technology needs to work when the vision model goes crazy because of the reflectivity of a surface or other such edge cases where it completely misunderstands where the object is.

whymauri• 2 years ago

I think LLMs provide more utility than further dependence on cars.

minwcnt5• 2 years ago

All modern AI is based on this, including self driving. Check out Waymo's research papers. They even have something called "Wayformer".

koala_man• 2 years ago

In Google's heyday, around 2014, I was talking with Uszkoreit about a possible role on his then NLP team.

I asked "What would you do if you had an unlimited budget?"

He simply said, "I do"

elevatedastalt• 2 years ago

Nice story, but Google's heyday was probably 10 years before that. By 2014 the decline had already started.

nostrademons2• 2 years ago

A little after that. I'd put the heyday between 2003-2010, starting with the GMail launch and ending with the Chrome & Android launches. That period includes GMail, Maps, Scholar, Orkut, Reader, the acquisitions of Blogger/YouTube/Docs/Sheets/Slides/Analytics/Android, Summer of Code, OAuth, Translate, Voice Search, search suggestions, universal search, Picasa, etc. Basically I can look at my phone or computer and basically everything I routinely use dates from that period.

OldMatey• 2 years ago

Agreed. I remember the town hall meeting where they announced the transition to being Alphabet. My manager was flying home from the US at the time. He left a Google employee and landed an Alphabet employee. I know it was probably meaningless in any real sense, but when they dropped the Dont Be Evil motto, it was a sign that the fun times were drawing to an end.

bigcat123456786• 2 years ago

Hahaha...

I worked at Borg

The quota system can kick in at whatever time the limits are reached.

And GPUs are scattered across borg cells, limiting the ceiling. That's why XBorg was created so that a global search among all Borg cells for researchers.

And data center Capex is around 5 billion each year.

Google makes hundres of billions of revenue each year.

You are asking what people would do in impossible situation. Like "what you do after you are dead", literally I could do nothing after I am dead.

I cannot even understand what I do stands for in the context of your question. The above is my direct reaction in the line that he assumes he had unlimited budget.

Jensson2• 2 years ago

I cannot even understand what I do stands for in the context of your question

That he had a higher budget than he knew what to do with. When I worked at Google I could bring up thousands of workers doing big tasks for hours without issue whenever I wanted, for me that was the same as being infinite since I never needed more, and that team didn't even have a particularly large budget. I can see a top ML team having enough compute budget to run a task on the entire Google scrape index dataset every day to test things, you don't need that much to do that, I wasn't that far from that.

At that point the issue is no longer budget but time for these projects to run and return a result. Of course that was before LLMs, the models before then weren't that expensive.

ShamelessC2• 2 years ago

Lighten up.

changoplatanero• 2 years ago

I shared an office with Uszkoreit when I was a phd intern and I always admired him for him having dropped out of his phd program.

lokar• 2 years ago

The HR system has (had?) “abd” as an education level/degree between MS and PhD

akprasad• 2 years ago

Those were fun times! (& great to see you again after all these years). It's astonishing to me how far the tech has come given what we were working on at the time.

ralph84• 2 years ago

And none of them still work for Google. It’s truly baffling that Google’s CEO still has a job after how badly he fumbled on AI.

hotstickyballs• 2 years ago

Just like Xerox and Bell. Innovation isn’t necessarily their most important job, at least when innovations are being made.

nemo44x• 2 years ago

He was too distracted with all the other things going on under his watch. He spent too much time trying to convince rank and file that it's OK to do business with customers they don't like, trying to ensure arbitrary hiring goals were made, and appeasing the internal activists that led protests and pressured to exit people that didn't fall in line.

This is why the CEO of Coinbase sent the memo a few years back stating that these types of people (not mission focused, self absorbed, and distracting) should leave the company. The CEO of Google should be fired.

tdb7893• 2 years ago

How has he been appeasing internal activists? He's been pretty clear about firing them

Edit: I'm also unclear on what makes you think he's spending much time on "arbitrary hiring goals". I remember an initiative for hiring in a wider variety, also conveniently much cheaper, locations but there wasn't indication he was personally spending time on it

calf2• 2 years ago

It is intellectually dishonest to call the military-industrial complex merely "customers that some Google employees didn't like".

Humans are more than their companies' mission, and your freedom to exercise your political views on how companies should be run inherently relies on this principle, too. So your argument is fundamentally a hypocrites projection.

0xcafecafe2• 2 years ago

Among other things (like cloud for instance). I always think Sundar is the Steve Ballmer of Google.

Thaxll2• 2 years ago

Well, ads are doing well.

paxys2• 2 years ago

Their wildest success at Google would have meant the company's stock price doubling or tripling and their stock grants being worth a couple million dollars at most. Meanwhile all of them had blank checks from top VCs in the valley and the freedom to do whatever they wanted with zero oversight. What exactly could Sundar have done to retain them?

pigscantfly• 2 years ago

It would have gone much better for Google if the Brain team had been permitted to apply their work, but they were repeatedly blocked on stated grounds of AI safety and worries about impacting existing business lines through negative PR. I think this is probably the biggest missed business opportunity of the past decade, and much of the blame for losing key talent and Google's head start in LLMs ultimately resides with Sundar, although there are managers in between with their own share of the blame.

pixl973• 2 years ago

I'm not 100% sure I agree...

Google is an ad company at the end of the day. Google is still making obscene amount of money with ads. Currently Gen AI is a massive expense in training and running and is only looking like it may harm future ad revenue (yet to be seen). Meanwhile OpenAI has not surpassed that critical threshold where they dominate the market with a moat and become a trillion dollar company. It is typically very hard for a company to change what they actually do, so much so that in the vast majority of cases its much more effective (money wise) to switch from developing products to becoming a rent seeking utility by means of lobbying and regulation.

Simply put, Google itself could not have succeeded with GenAI without an outside competitor as it would have become its own competition and the CEO would have been removed for such a blunder.

moralestapia• 2 years ago

it may harm future ad revenue (yet to be seen)

It harms ad revenue at the moment.

For some specific queries (like code-related), I now go to chat.openai.com before google.com, I'm sure I'm not the only one.

AI is the only thing that can put an end to Google's dominance of the web. I also have no idea how Sundar still has a job there.

CuriouslyC• 2 years ago

Google is an AI/ML company that has shoehorned AI/ML researchers into working on ads to pay the bills.

pchristensen• 2 years ago

Classic Innovator's Dilemma case study.

com2kid2• 2 years ago

It is typically very hard for a company to change what they actually do

Microsoft started out selling BASIC runtimes. Then they moved into operating systems. Then they moved into Cloud. Satya seems to be putting his money where his mouth is and working hard to transition the company to AI now.

Apple has likewise undergone multiplet transformations over the decades.

However Google has the unique problem that their main cash cow is absurdly lucrative and has been an ongoing source of extreme income for multiple decades now, where as other companies typically can't get away with a single product line keeping them on top of the world for 20+ years. (Windows being an exception, maybe Microsoft is an exception, but say what you will about MS, they pushed technology forward throughout the 90s and accomplished their goal of a PC in every house)

xNeil• 2 years ago

It's not about just ad revenue though, is it? Many thousands, if not millions of sites rely entirely on Google Search for their revenue. You cannot simply replace all of them with a chatbot - there are massive consequences to such decisions.

ryandvm• 2 years ago

I think the issue is that there is no future for a trustworthy AI that doesn't completely cannibalize their ad revenue cash cow.

Like, who wants to use an AI that says things like, "... and that's why you should wear sunscreen outside. Speaking of skin protection, you should try Banana Boat's new Ultra 95 SPF sunscreen."

HarHarVeryFunny4• 2 years ago

Yeah, but isn't the idea to cannibalize your own products before someone else does?!

In any case, consumer chatbots isn't the only way to sell the tech. Lot's of commercial use too.

I don't see why ads couldn't be integrated with chatbots too for that matter. There's no point serving them outside of a context where the user appears to be interested in a product/service, and in that case there are various ways ads could be displayed/inserted.

kccqzy• 2 years ago

You are not imaginative enough.

open592• 2 years ago

"Based on the research I have available wearing sunscreen with a SPF at or greater than 95 is the most likely to prevent the harmful effects of sunburn".

Go to google -> Search "sunscreen" -> (Sponsored) Banana Boat new Ultra 95 SPF sunscreen

Oh wow! 95 SPF! Perfect

acdha• 2 years ago

Like, who wants to use an AI that says things like, "... and that's why you should wear sunscreen outside. Speaking of skin protection, you should try Banana Boat's new Ultra 95 SPF sunscreen."

On the other hand, their history suggests most people would be fine with an AI which did this as long as it was accurate:

... and that's why you should wear sunscreen outside.
Sponsored by: Banana Boat's new Ultra 95 SPF sunscreen…"

seatac76• 2 years ago

I mean if you had the pedigree these people have, the right move is to leave a big company and go start your own thing, there is simply a lot more upside there. You’d be foolish to stay with a big company, not sure what Google could have done here. Not everyone wants to be Jeff Dean.

mzs• 2 years ago

Google doesn't have to try to be the best at AI now, it can just buy whoever is the winner later - a lesson learned from google video.

robocat• 2 years ago

No Google can't just buy the winner.

Google purchased YouTube for $1.65 billion (and I recall that seemed overpriced at the time).

The way VC works for private companies now is completely different than in 2006.

vlakreeh• 2 years ago

If that winner turns out to be OpenAI and they attempt to buy them it'll be blocked by regulators instantly.

Thaxll2• 2 years ago

It's pretty crazy to think that Google is not OpenAI today, they had deep mind and an army of PHDs early on.

dpflan2• 2 years ago

What is OpenAI today? Can you elaborate? Google is a varied $trillion company. OpenAI sells access to large generative models.

bitcharmer• 2 years ago

Sure, Google is worth more but they essentially reduced themselves to an ad business. They aren't focused on innovation as much as before. Shareholder value is the new god.

bevekspldnw3• 2 years ago

Where would most people want to work is the question. This is all a talent game.

password54321• 2 years ago

Well at the time before Microsoft got involved it was sort of an unspoken rule among the AI community to be open and not release certain models to the public.

sharemywin2• 2 years ago

I think that was because it become a product that point.

pests2• 2 years ago

To be open and not release? Isn't that at odds?

cgearhart• 2 years ago

I think it’s evidence that timing is everything. In the 2010s deep learning was still figuring out how to leverage GPUs. The scale of compute required for everything after GPT-2 would have been nearly impossible in 2017/2018–our courses at Udacity used a few hours of time on K80 GPUs. By 2020 it was becoming possible to get unbelievable amounts of compute to throw at these models to test the scale hypothesis. The rise of LLMs is stark proof of the Bitter Lesson because it’s at least as much the story of GPU advancement as it is about algorithms.

empath-nirvana2• 2 years ago

The problem is that chatting with an LLM is extremely disruptive to their business model and it's difficult for them to productize without killing the golden goose.

yobbo• 2 years ago

Yes, but Google's search volume would make it impossible either way.

But they are probably using various techniques in analyses, embeddings, "canned" answers to queries, and so on.

crowcroft3• 2 years ago

I know everyone cites this as the innovators dilemma, but so far the evidence suggests this isn't true.

ChatGPT has been around for a while now, and it hasn't led to a collapse in Google's search revenue, and in fact now Google is rushing to roll out their version instead of trying to entrench search.

A famous example is the iPhone killing the iPod, and it took around 3 and a half years for the iPod to really collapse, so chat and co-pilots might be early still. On the other hand handheld consumer electronics have much longer buying cycles than software tools.

solomonb• 2 years ago

Sounds like Xerox all over again.

thehappypm• 2 years ago

No, I think very few people truly believed that a souped-up SmarterChild would be all that interesting. Google focused more on winning board games.

bevekspldnw3• 2 years ago

Ironically they killed the golden goose (the web) with display ads which incentivized low quality content to keep users scrolling. To put salt in their own wound, they are now indexing ever more genAI crap than they can handle and search is imploding.

mike_hearn• 2 years ago

No, it's the other way around. Right now web search is "disruptive" to the LLM business model.

Google is a business fundamentally oriented around loss leaders. They make money on commercial queries where someone wants to buy something. They lose money when people search for facts or knowledge. The model works because people don't want to change their search engine every five minutes, so being good at the money losing queries means people will naturally use you for the money making queries too.

Right now LLMs take all the money losing queries and spend vast sums of investor capital on serving them, but they are useless for queries like [pizza near me] or [medical injury legal advice] or [holidays in the canary islands]. So right now I'd expect actually Google to do quite well. Their competitor is burning capital taking away all the stuff that they don't really want, whilst leaving them with the gold.

Now of course, that's today. The obvious direction for OpenAI to go in is finding ways to integrate ads with the free version of ChatGPT. But that's super hard. Building an ad network is hard. It takes a lot of time and effort, and it's really unclear what the product looks like there. Ads on web search is pretty obvious: the ads look like search results. What does an ad look like in a ChatGPT response?

Google have plenty of time to figure this out because OpenAI don't seem interested. They've apparently decided that all of ChatGPT is a loss leader for their API services. Whether that's financially sustainable or not is unclear, but it's also irrelevant. People still want to do commercial queries, they still want things like real images and maps and yes even ads (the way the ad auction works on Google is a very good ranking signal for many businesses). ChatGPT is still useless for them, so for now they will continue to leave money on the table where Google can keep picking it up.

scottyah• 2 years ago

They'd have to fall pretty far to be OpenAI.

freddealmeida• 2 years ago

Attention models? Attention existed before those papers. What they did was show that it was enough to predict next word sequences in a certain context. I'm certain they didn't realize what they found. We used this frame work in 2018 and it gave us wildly unusual behavior (but really fun) and we tried to solve it (really looking for HF capability more than RL) but we didn't see what another group found: that scale in compute with simple algorithms were just better. To argue one group discovered and changed AI and ignore all the other groups is really annoying. I'm glad for these researchers. They deserve the accolade but they didn't invent modern AI. They advanced it. In an interesting way. But even now we want to return to a more deterministic approach. World models. Memory. Graph. Energy minimization. Generative is fun and it taught us something but I'm not sure we can just keep adding more and more chips to solve AGI/SGI through compute. Or maybe we can. But that paper is not written yet.

calepayson• 2 years ago

I’m studying neuroscience but very interested in how ai works. I’ve read up on the old school but phrases like memory graph and energy minimization are new to me. What modern papers/articles would you recommend for folks who want to learn more?

voiceblue2• 2 years ago

For phrases, Google's TF glossary [0] is a good resource, but it does not cover certain subsets of AI (and more specifically, is mostly focused on TensorFlow).

[0] https://developers.google.com/machine-learning/glossary

peppertree• 2 years ago

If you are in neuroscience I would recommend looking into neural radiance fields rendering as well. I find it fascinating since it's essentially an over-fitted neural network.

svachalek• 2 years ago

Someone put this link up on another discussion the other day and I found it really fascinating:

https://bbycroft.net/llm

I believe energy minimization is literal, just look at the size of that thing and imagine the power bill.

arcen• 2 years ago

He is most likely referring to some sort of free energy minimization.

voiceblue2• 2 years ago

but we didn't see what another group found: that scale in compute with simple algorithms were just better

The bitter lesson [0] strikes again.

[0] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

tracerbulletx• 2 years ago

I kind of wonder if the reason this seems to be true is that emergent systems are just able to go a lot farther into much more complex design spaces than any system a human mind is capable of constructing.

fnordpiglet• 2 years ago

In fairness, an article that’s about “the Google engineers who incrementally advanced AI” wouldn’t sell many advertisements.

acchow• 2 years ago

I guess the same thing happened with special relativity and Poincaré was forgotten.

autokad3• 2 years ago

I argue they definitely 'changed AI' but definitely agree they didn't 'invent modern AI'.

Personally, I think both compute and NN architecture is probably needed to get closer to AGI.

godelski• 2 years ago

This is a classic... This isn't the first time this piece has been posted before either. Here's one from FT last year[0] or Bloomberg[1]. You can find more. Google certainly played a major role, but it is too far to say they invented it or that they are the ones that created modern AI. Like Einstein said, shoulder of giants. And realistically, those giants are just a bunch of people in trench coats. Millions of researchers being unrecognized. I don't want to undermine the work of these researchers, but that doesn't mean we should also undermine the work of the many others (and thank god for the mathematicians who get no recognition and lay all the foundation for us).

And of course, a triggered Yann[2] (who is absolutely right).

But it is odd since it is actually a highly discussed topic, the history of attention. It's been discussed on HN many times before. And of course there's Lilian Weng's very famous blog post[3] that covers this in detail.

The word attention goes back well over a decade and even before Schmidhuber's usage. He has a reasonable claim but these things are always fuzzy and not exactly clear.

At least the article is more correct specifying Transformer rather than Attention, but even this is vague at best. FFormer (FFT-Transformer) was a early iteration and there were many variants. Do we call a transformer a residual attention mechanism with a residual feed forward? Can it be a convolution? There is no definitive definition but generally people mean DPMHA w/ skip layer + a processing network w/ skip layer. But this can be reflective of many architectures since every network can be decomposed into subnetworks. This even includes a 3 layer FFN (1 hidden layer).

Stories are nice, but I think it is bad to forget all the people who are contribution in less obvious ways. If a butterfly can cause a typhoon, then even a poor paper can contribute to a revolution.

[0] https://www.ft.com/content/37bb01af-ee46-4483-982f-ef3921436...

[1] https://www.bloomberg.com/opinion/features/2023-07-13/ex-goo...

[2] https://twitter.com/ylecun/status/1770471957617836138

[3] https://lilianweng.github.io/posts/2018-06-24-attention/

a_wild_dandan• 2 years ago

This is an uncharitable and oddly dismissive take (i.e. perfect for HN, I suppose).

Today's incredible state-of-the-art does not exist without the transformer architecture. Transformers aren't merely some lucky passengers riding the coattails of compute scale. If they were, then the ChatGPT app which set the world ablaze would've instead been called ChatMLP, or ChatCNN. But it's not. And in 2024 we still have no competing NLP architecture. Because the transformer is a genuinely profound, remarkable idea with remarkable properties (e.g. training parallelism). It's easy to downplay GPTs as a mostly derivative idea with the benefit of hindsight. I'm sure we'll perform the same revisionist history with state-space models, or whatever architecture eventually supplants transformers. Do GPTs build on prior work? Do other approaches and ideas deserve recognition? Yeah, obviously. Like...welcome to science. But the transformer's architects earned their praise -- including via this article -- which isn't some slight against everyone else, as if accolades were a zero-sum game. These 8 people changed our world and genuinely deserve the love!

hn_throwaway_99• 2 years ago

Question for you, as someone relatively new to the world of AI (well, not exactly new - I took many courses in AI, including neural networks, but in the late 90s... the world is just a tad different now!)

Is there any good summary of the history of AI/deep learning from, say, late 00s/2010 to the present? I think learning some of this history would really help be better understand how we ended up at the current state of the art.

cooljoseph• 2 years ago

And in 2024 we still have no competing NLP architecture.

No, we do. State space models are both faster and scale just as well. E.g., RWKV and Mamba.

Transformers aren't merely some lucky passengers riding the coattails of compute scale.

Err... they are, though. They were just the type of model the right researchers were already using at the time, probably for translating between natural languages.

goertzen• 2 years ago

It’s reply’s like this that make HN a good place.

j7ake• 2 years ago

They didn’t invent it, they advanced it. I agree.

However, some advances can have huge consequences to the field compared to others, even if at the technical level they appear comparable.

One example that comes to mind is CRISPR.

xnx• 2 years ago

Not only were the authors all Google employees, they also worked out of the same offices.

Subtle plug for return-to-office. In-person face-to-face collaboration (with periods of solo uninterrupted deep focus) probably is the best technology we have for innovation.

tarsinge• 2 years ago

with periods of solo uninterrupted deep focus

Which is usually impossible in the office. So more like a mix, which is what all reasonable people are saying.

kick_in_the_dor• 2 years ago

+1. I worked remotely the past 2 years. I just RTO'd (new team) and it's life-changing when you actually want to get stuff done quickly.

bevekspldnw3• 2 years ago

“Office” does not have to mean open office. Academics all have offices with doors for a reason. I can’t stand open office, but private office in a building with other people is great.

steven• 2 years ago

Not so subtle!

throw0101d• 2 years ago

This 'conversation' with Geoffrey Hinton and Fei-Fei Li goes over a lot of the history of things (1h50m):

* https://www.youtube.com/watch?v=QWWgr2rN45o

* https://www.youtube.com/watch?v=E14IsFbAbpI ('mirror')

Goes over Hinton's history and why he went the direction he did with his research, as well as Li's efforts with ImageNet.

lordswork3• 2 years ago

It's pretty cool that Google employees can see the first transformer implementation (and the reviewer comments) in cs/.

So many monumental moments in AI history are archived in Google's intranet.

tinyhouse• 2 years ago

It's amazing how much industry changed CS in the last decade. Significantly more than Academia. Resources is part of it of course, but I also think that the majority of best people in CS choose research labs in industry over academia. No surprise here of course - if openai offers you more than $500K a year, why would you go to academia, make peanuts in comparison and worry about politics for your tenure? Then when they get bored at big tech they raise money and start a company.

nojvek• 2 years ago

Google is an Ads company ad the end of the day. They are making 10% revenue from GCloud but still bulk of their profits are from Ads.

If they were threatened, Google could have easily owned the cloud. They were the first to a million servers and exabyte storage scale. Their internal infra is top notch.

Under no circumstances can they damage their cash cow. They have already shown to be UX hostile by making Ads look like results, and Youtube forcing more ads onto users.

The biggest breakthrough an AI researcher can have at Google is to get more ad clicks across their empire.

Google management knows this. Without ad revenue, they are fucked.

That is my biggest fear. That with AI we are able to exploit the human mind for dopamine addiction through screens.

Meta has shown that their algorithm can make the feeds more addictive (higher engagement). This results in more ads.

With Insta Reels, Youtube, TikTok e.t.c "Digital Heroin" keeps on getting better year after year.

poorman• 2 years ago

I'm getting tired of the "Google employee who created X". There is so much prior art it makes me roll my eyes when I see these posts.

victor106• 2 years ago

The group is also culturally diverse. Six of the eight authors were born outside the United States; the other two are children of two green-card-carrying Germans who were temporarily in California and a first-generation American whose family had fled persecution, respectively.

As much as I think the America has lot of things it needs to fix, there is no other country on earth this would be possible. That's just a fact.

yeshellohi• 2 years ago

there is no other country on earth this would be possible. That's just a fact

I don’t think this the case. If anything the US makes life very hard for even high-skilled work-based immigrants. Many countries have a higher % of foreign born residents than the US (Singapore, Australia, Germany, Canada)

I myself used to work at Google UK and my own team was 100% foreign born engineers from every continent.

malshe• 2 years ago

I agree. As someone who has lived or worked in Asia, Europe, and the US, I always had the most diverse group of colleagues in the US.

The Google employees who created transformers

Related Stories