Ex-Googlers raise $40M to democratize natural-language AI

https://www.fastcompany.com/90670635/ex-googlers-raise-40-million-to-democratize-natural-language-ai

by aaronbrindle • 5 years ago

201 121 5 years ago

samstave3• 5 years ago

I have been thinking recently about the ethics of AI;

At what point do we 'allow' AI to determine actions based on perceived (programmed) *BIAS* How can one prevent any bias on an AI's ability to be deterministic.

analyst74• 5 years ago

AI simply replicate existing bias the data already has. The downside is that it could amplify the bias if not done carefully; the upside is that now we can analyze the algorithm's bias and fix it, which is much harder to do to people with biases.

fallingknife• 5 years ago

But to do this you would need unbiased people. Since those don't exist, this correction would just be matching the bias to that of the bias adjuster.

taneq• 5 years ago

All of the conversations I've seen about AI bias recently seem to define "bias" as "any difference between the output and the particular rightthink ordained by whoever's speaking." Nobody cares about making the AI's output correct, they just want it to agree with them.

So the glib answer is "train it on unbiased data". Depending on your philosophy, this translates either to "manually 'fix' anything you see as 'bias' in the training data", or "use a sufficient amount of entirely unmodified raw data along with an algorithm sufficiently insightful to cancel out all of the sources of inaccuracy introduced by the various sources of data and extract ground truth."

But you know if anyone ever does manage the latter, its results will still be decried as 'biased' by anyone who disagrees with them.

trhway• 5 years ago

Nobody cares about making the AI's output correct, they just want it to agree with them.

that is the 4th industrial revolution, the "post-correct" world where abundance of information (and its consumers) and speed of its production allows for and results in co-existence of multiple truths (kind of like hyperbolic geometry where one can draw through a point multiple different lines parallel to the given line) with the information space splitting into multiple feudal times like dukedoms.

Anyway, in general for biases i think we have D.Rumsfeld situation - the known/expected biases are known while the AI driven world would most probably bring new biases that we don't even expect.

samstave3• 5 years ago

I am now convinced this is what is meant by this:

https://en.wikipedia.org/wiki/Ouroboros

but instead its the warning against letting AI iterate upon itself without external intervention...

OneEyedRobot• 5 years ago

What if an AI is unbiased but makes decisions people don't like?

monkeyfacebag• 5 years ago

Is "democratize" the hip way to spell "sell" these days?

xkeysc0re• 5 years ago

Vote with your dollar!

totetsu• 5 years ago

I've got some old clothes I'm planning to take to take to the recycle shop and Democratize over the weekend.

vmception• 5 years ago

Tuesdays are buzzword bingo nights

LdSGSgvupDV• 5 years ago

And "AI".

amelius• 5 years ago

Let's democratize our democracy to the highest bidder!

timoth3y• 5 years ago

"democratize"

You keep using that word. I do not think it means what you think it means.

The word you are looking for is "mass-market" or perhaps "sell to small business".

OnlineGladiator• 5 years ago

I think the word is monetize.

qPM9l3XJrF• 5 years ago

Or, for a darker spin, "absolve yourself of responsibility if anyone misuses your API". I actually think OpenAI gets this right with GPT-3, having fairly strict terms of use for their API, so I'm sad to see seemingly less responsible players enter the space.

robbedpeter4• 5 years ago

EleutherAI won this fight already. The technology is freely available for anyone to use however they like, and they're responsible for whatever it is they do with it.

The 6 billion parameter model was released just last week with the huggingface api hooks. It's just a few percentage points less performant than Gpt-3 DaVinci in most metrics. You can run it on a laptop with 40gb ram, albeit slowly.

All that to say, the horse has left the barn. What openai is doing is just marketing and protecting an investment. They don't have an ethical or moral high ground.

rahimnathwani3• 5 years ago

What they say they're doing fits one of the definitions of democratize: "make (something) accessible to everyone".

More about the etymology here: https://www.etymonline.com/word/democratize

8note• 5 years ago

Commoditize as an alternative?

dna_polymerase• 5 years ago

All that source tells us is that the word has been wrongly used since the french revolution. Democracy means governance by the masses. Has nothing to do with making things accessible to the masses.

abeppu2• 5 years ago

Which is more "democratized": a large language model which can be downloaded, and accessed as a library, originally based from the work of the FAANG giants (e.g. huggingface transformers) or an API, where every invocation is a call that flows through Cohere's servers?

vgeek• 5 years ago

In America, money is speech-- especially in politics, so unfortunately maybe this is a "fitting" use of the word.

vkou• 5 years ago

"Democratized" in the context of a startup usually doesn't mean letting the public access something, it usually means letting a different group of investors (the funders of the startup) access a market (formerly controlled by a monopoly or oligopoly of established mega-firms).

samhw4• 5 years ago

it usually means letting a different group of investors (the funders of the startup) access a market

I guess 'democratize' rolls off the tongue more easily than 'heteroligocratize'.

BoxOfRain3• 5 years ago

This reminds me of the whole privatisation vs nationalisation debates in the UK. Labour claimed when it nationalised the railways and other parts of the economy it was giving them "back to the people" in a democratic fashion because they were now owned by the state which it conceives of as an expression of the body politic, rather than corporations which are seen as entirely seperate from the people. The Tories when they privatised these things also claimed they were giving them "back to the people" in a democratic fashion as private individuals could now freely invest if they chose in these companies rather than them being controlled by the state, which it conceives of as something entirely seperate from the people*.

"Democracy" and especially "the people" can mean lots of different and completely contradictory things to lots of different people. I'm always very cautious of anyone who invokes "democracy" and "the people" directly, I don't automatically consider them untrustworthy but I prefer a direct argument for a particular idea. If someone's truly a democrat, they'd be unafraid to make their point without such potentially dishonest tactics as people would democratically choose that idea anyway.

* I'm massively oversimplifying here - there's such things as Blairism and One-Nation Conservatism which blur these lines enormously.

1vuio0pswjnm7• 5 years ago

The former.

"Data dumps" are privacy-friendly. For example, a user can download Wikipedias dumps and search through them to her hearts content, and never use the network. Zero data collection by third parties. All those observing the network can see is that she downloaded some data dumps.

"Web APIs" are "tech" company and surveillance-friendly. Network access is required and all activity is observed and recorded. Web APIs are also used as a means of controlling access to what is often publicly available data/info. The company does not own the data/info, its a middleman. "Too many" requests, API user gets cut off.

Too often its publicly available data/info that is being served by APIs. Hard to sell data dumps of public data as a "product". I sometimes see entities that provide "data dumps", e.g., a corpus, for free who then try to restrict usage of it through a license, yet they do not themsleves own the data. Whether they even have the rights to "license" it is debatable. They are never legally challenged so we cannot say for sure. The more interesting issue is whether they had the rights to collect the data thats in it. The so-called "web scraping" issue.

IIAOPSW• 5 years ago

Reminds me of a joke.

"They say New Yorkers are selfish and unfriendly, but its all untrue. When I visited, a guy overheard I was a tourist and came right up and offered me a great deal on Staten Island Ferry tickets. Just $7.50 for a round trip!"

(In case you don't know, the Staten Island Ferry is free).

yobbo• 5 years ago

Not only that, democratize would imply that the general public has influence on the project through voting, for example.

BoxOfRain3• 5 years ago

I think there's degrees of democracy. There's the directly democratic model at one end and "not hidden away in a black box at $mega_corp" on the other. You can be in favour of democracy without advocating mob rule for example.

tgsovlerkhgsel• 5 years ago

If the former has equivalent performance (as in output quality) etc., then the former obviously.

If the former doesn't exist (at a comparable quality), then the latter is better than the nothing.

beaconstudios3• 5 years ago

When a startup says anything, you should hear "we plan to use this to get hella rich".

There is no such thing as a startup with altruistic intent.

moonchrome• 5 years ago

Altruistic startup = a non profit ?

WanderPanda2• 5 years ago

But it is also not zero sum. Long term viable companies provide a higher value to their customers than the price they charge.

pinky1417• 5 years ago

To reductio ad absurdum that - every time you see any marketing, eg “Drink Coca-Cola because it’s refreshing”, you should hear “Drink Coca-Cola because it’ll make us money.”

To your point though: “democratize x” makes my eyes roll. It’s overused hip marketing speak.

jasode2• 5 years ago

Which is more "democratized": a large language model which can be downloaded, and accessed as a library, [...] or an API, where every invocation is a call that flows through Cohere's servers?

I do understand the point you're trying to make: local autonomy is superior to cloud access mediated by a private commercial entity.

However at this time, we may have a counterintuitive situation where API access is more "democratic" than downloading a huge model.

Based on various reports[1], the GPT-3 model was trained on ~45 terabytes of text corpus (Wikipedia + Web Common Crawl + book texts, etc) and the final runtime model (175 billion parameters) requires ~350 gigabytes of RAM. In that case, the model size is ~1% of the training set.

So "democractize" depends on how ambitious the user is. If you want to use a very large 350GB RAM model, the cloud model with API will be more accessible by the masses than running on local hardware. Last time I looked, an Intel Xeon motherboard has max ram of 128GB so scaling up to 350GB RAM is not going to be cheap or trivial to build.

Let's further extrapolate to a future hypothetical GPT-4 using ~10x multiplier: train on 450 terabytes of text with a model requiring 3.5 terabytes of RAM. How do we make that future huge model accessible to the masses? Probably via a cloud API. Unfortunately, there's an unavoidable hardware capital expense barrier there.

[1] https://www.google.com/search?q=gpt-3+350gb+ram

occamrazor• 5 years ago

If OpenAI made the weights and the NN configuration freely available,I am sure that several organizations would offer cheap (or even free with strict rate limits) API access. And users would also be able to run the model locally, if they can afford the hardware.

WanderPanda2• 5 years ago

This is a strawman argument. Publishing the code/weights is not mutually exclusive to providing an API

sciurus• 5 years ago

Last time I looked, an Intel Xeon motherboard has max ram of 128GB so scaling up to 350GB RAM is not going to be cheap or trivial to build.

Even a low-end tower server can be configured with multiple terabytes of RAM.

E.G. https://www.dell.com/en-us/work/shop/povw/poweredge-t640#tec...

BoxOfRain3• 5 years ago

What's wrong with both? A true democratisation of this kind of model would involve both an offline model for those with the resources to support such a thing as well as many different people hosting them and allowing access for a price via an API for those without those high end resources.

I think the issue people have with centralised cloud APIs for this sort of thing is that there's still a single gatekeeper with their finger on the off switch. In my opinion, instead of throwing out cloud APIs altogether a better scenario would be many gatekeepers with a deliberate diversity of socio-political backgrounds.

greens• 5 years ago

This is a common misconception. GPT-3 was trained using a 300B token (~300gb) subset of common-crawl and friends. The model is larger than the dataset.

shamoo• 5 years ago

What stops Google, Microsoft, etc from doing the same if Cohere's solution gains traction?

abeppu2• 5 years ago

Don't they try this currently? https://cloud.google.com/natural-language/docs

I think during a gold rush, it's good to sell shovels even if can't be the leading shovel sales outlet in town.

I think we're at a point where new NLP techniques will create a lot of value, but it's still hard to tell in advance which cases the current techniques can do well enough to be worthwhile, and in which cases they'll be kinda cool but not effective and so we'll continue to require humans to be involved catching errors. While we sort that out, a lot of companies will need to take a few stabs at trying to get some new model to work for their problems.

robbedpeter4• 5 years ago

Bureaucratic inertia, sunk cost fallacy, interdepartmental resource hoarding, corporate bloat, and finally, HR and middle management all jealously guarding their petty kingdoms.

Oh, and the fact that Cohere is an expensive solution for a problem that's already been solved for free by open source groups.

twelfthnight2• 5 years ago

Are you talking about https://huggingface.co/EleutherAI/gpt-neo-1.3B?

DenisM2• 5 years ago

I wonder if either of the two companies have introduced many new products recently?

When you only have $1m you might as well poke around various ideas and see if you can get another million or ten. Now, if you're sitting on tens of billions of dollars your priorities change - it's far more profitable to grow a $10b pie by 10% than a $1m by 1000%. I'm pretty sure that's what happened to Oracle and IBM, hence we rarely hear about them anymore.

exdsq• 5 years ago

What does democratize even mean?

SpicyLemonZest• 5 years ago

It's a trendy marketing word for selling to a lot of smaller businesses, in contrast with "normal" enterprise software sales which stereotypically focus on big contracts.

tdeck• 5 years ago

I think this kind of marketing language is aimed more at potential employees / investors than customers. Employees and investors often want to imagine themselves as part of a narrative where they're "democratizing" something.

rahimnathwani3• 5 years ago

Here, it means 'make available to everyone'.

It has at least one other meaning ('introduce democracy as a system of rule').

robbedpeter4• 5 years ago

Like synergize, but with 20% more righteous and warm fuzzy.

DenisM2• 5 years ago

de·moc·ra·tize /dəˈmäkrəˌtīz/

verb

introduce a democratic system or democratic principles to. "public institutions need to be democratized"

make (something) accessible to everyone. "mass production has not democratized fashion"

https://www.google.com/search?q=democratize

I suppose that would be the second meaning.

CuriousCosmic• 5 years ago

The real answer: Democratisation of software or systems is making Free and Open Source all the software and knowledge required to operate the software or system. Democratisation is also implementing structures that make the development process beholden and accountable to the users. Fundamentally it's about giving the users true ownership over the software and a voice in the development process and/or operation.

Now how it's used here? It's more just a buzzword that suggests they are trying to make it more accessible by selling it as a SaaS. They seem to have a focus on ethically providing access to these more powerful ML systems but what this'll actually mean beyond "cover our ass" is up in the air.

blahblahblogger• 5 years ago

"Democratize".

lol. It's now a weasel word used by start-ups. Merits an eye roll before moving along.

jollybean• 5 years ago

Wonderful concept but whenever I hear the term 'democratize' in a description I laugh a little inside. It's hugely overused, to the point where lazy marketers of all kinds are using it willy nilly.

We need better lingo.

mavsman• 5 years ago

Just watched the WeWork documentary last night and it makes me question the validity of every money raise now.

pathseeker• 5 years ago

"democratize" is double-speak for "closed source".

TeMPOraL• 5 years ago

Worse, it's double-speak for SaaS APIs, aka. "you have to enter into a business relationship with us in order to use it".

zitterbewegung• 5 years ago

This seems much more developed than whatever is being done here:

https://huggingface.co/transformers/index.html

I recommend it since they already are democratizing AI.

ceocoder• 5 years ago

They should call it something that makes it clear how “democratic” they are and the fact they are in AI. Hmmm, like Open but about AI. I give up.

throwawaylinux• 5 years ago

Democratizing AI is passe. If they have no plans to decolonize AI then they're part of the problem.

1_over_n• 5 years ago

fast.ai are doing the best work democratizing AI (machine learning and neural networks), including natural language

Can't recommend the course enough and it's incredible that such a quality resource is available for essentially zero cost

https://github.com/fastai/course-nlp

https://www.youtube.com/playlist?list=PLtmWHNX-gukKocXQOkQju...

elliekelly• 5 years ago

The “Why is this course taught in such a weird order?” section is really interesting.

Tenoke2• 5 years ago

I am/was a huge fan and have done their courses but they've made so many weird decisions (like going all in on swift4tf for teaching newish people so early on or the dramas they involve themselves in) and have stopped doing as much that I'm not sure if they are as great of a current example anymore (but maybe they'll resurface stronger).

Huggingface are a better example closer to what Cohere do.

root_axis• 5 years ago

"Commoditize" not "democratize". If it were democratized I'd be able to spin up my a local instance of their API. With the setup described in the article they are the unilateral arbiters of access to and usage of the technology.

joe_the_user• 5 years ago

Large language model are the rage, it seems.

Can someone a successful (not necessarily profitable) concrete application of these things, other than "gpt-3 wrote an article in Guardian and said it wouldn't kill us".

forgotmypw17• 5 years ago

In the narrow access window I was allowed to Philosopher AI, I found it incredibly helpful in brainstorming and bouncing ideas off. It helped me organize my project, and I even included the conversations in the repo.

ivalm• 5 years ago

We use specifically prompted gpt-3 to generate synthetic training examples (eg paraphrases, summaries, etc). We fine tune other (much smaller than gpt3 but still large-ish) language models for controllable language generation (often augmented with synthetic data from gpt3). As a comparison point, we did try GPT Neo and it did not provide sufficiently high quality synthetic data.

Transformers in general have lots of applications (machine translation, information retrieval/reranking, ner, etc).

nl• 5 years ago

I used largish (GPT-2 and similar) models to build an app discovering Category Entry Points (a marketing thing around the things people are thinking about when they decide they need to buy a particular product) for specific product categories.

It was very successful.

sanxiyn• 5 years ago

GitHub Copilot?

iaml• 5 years ago

AI Dungeon?

Olshansky• 5 years ago

The same day this made the front page of hacker news, one of the founders is also playing his first live concert with his band tonight: https://twitter.com/goodkidband/status/1435267485683568642.

The best part of the story for me is the work-life-balance he's able o achieve.

guiambros• 5 years ago

Ha, what a great find. First time listening, but liked their music [1].

[1] https://www.youtube.com/watch?v=2t9TxR4IolM

mostelato• 5 years ago

All of us in the band are engineers (I work at Snapchat) and we take the band super seriously and make time for it

ergot_vacation• 5 years ago

In the last year, critics of large NLP models, which are trained on huge amounts of text from the web, have raised concerns about the ways that the technology inadvertently picks up biases inherent to the people or viewpoints in this training data. Such critiques gained steam after Google controversially pushed out famed AI researcher Timnit Gebru, in part due to a paper she coauthored analyzing these risks. Cohere CEO Aidan Gomez says his company has developed new tools and invested a lot of time into making sure the Cohere models don’t ingest such bad data.

So it's censored. No thanks, not interested.

pjfin123• 5 years ago

Argos Translate has open source neural machine translation https://github.com/argosopentech/argos-translate

andrew_• 5 years ago

"Democratize" has got to be the most abused new buzzword in tech for this decade. Right up there with "synergy" from yesteryear.

fragmede• 5 years ago

Cloud seems like it should be between those buzzwords, chronologically.

felipellrocha• 5 years ago

Is Ex-Googler still a thing when everyone is an Ex-Googler?

shados• 5 years ago

I was about to post that. Anyone who reads the little PDF Google gives you when you apply there knows pretty much exactly how the interview will go, and the questions they frequently ask are all over the internet. At that point its just about cramming your brain for a couple of days.

Like, I don't know how to write an A* on top of my head, but back when I was looking at Google a while ago, they always asked that question, and the prep PDF said so. Kind of hard to fail there...

Not everyone will be able to work there, but it's certainly not the status symbol it was a long time ago. Plenty of "Ex Googlers" are code monkeys like anyone else from any other company.

kovac• 5 years ago

Haha, I see this even on some people's dating profiles ¯\_(ツ)_/¯

j_barbossa• 5 years ago

Nothing is democratized here, this is pure marketing.

Foobar8568• 5 years ago

When your product revolves around that you are an ex-Googler...

mnd999• 5 years ago

It’s not like it’s even verifiable is it? Literally anyone can say that.

___luigi4• 5 years ago

In the ML industry, there are two type of companies.

1/ Open source companies like huggingface (https://github.com/huggingface), explosion (https://github.com/explosion), Fast.ai that democratized access to ML and provided a set of tools & language models for engineers. These companies didn't only manage to build a company and open source projects, they built a welcoming community and it's impressive to see all these students use these tools to tackle bigger problems. That's what"democratized" mean to an engineer[1].

2/ Media companies that built ML APIs to ease the use of such services like OpenAI. I think this company falls under this category.

[1] https://marksaroufim.substack.com/p/machine-learning-the-gre...

fxtentacle3• 5 years ago

Do you know how Huggingface can pay their bills? How do you monetize an AI that everyone can freely download?

___luigi4• 5 years ago

Yes. HF is built on a tiered subscription (SaaS) model. https://huggingface.co/pricing

You can refer to this discussion for more details: https://twitter.com/migueldeicaza/status/1285204129225281536

streetcat1• 5 years ago

Unfortunately the new models are very big, hence they cannot be released as smaller models (as open source). They must be served as an api.

___luigi4• 5 years ago

.. They must be served as an api.

I respectfully disagree with this statement, there is a large body of work focusing on making this easier. The HW/SW stack vary from a a company to another.

d13• 5 years ago

GPT-J is huge and completely open source

fxtentacle3• 5 years ago

How does the business model work here?

For a language processing SDK to be useful, it needs to work inside an offline capable app on my phone. But doesn't that make it way too easy for someone to extract the network and use it to train their "own" AI with distillation learning?

And if they instead only run the AI on their servers so users need to connect through an API, how is that any different from the language AI APIs that Google, Amazon, Microsoft already offer?

And let's say they foot the bill to train a new type of language AI, what stops the big cloud providers from just training something similar? A 200mio non-recurring expense won't stop Google if you have proven that it's a viable business for you.

Asking because I'm pondering similar issues for my AI project.

samhw4• 5 years ago

But doesn't that make it way too easy for someone to extract the network and use it to train their "own" AI with distillation learning?

If you're targeting medium-to-large businesses, I suspect lots of them will buy a licence just to make sure they can't be sued (e.g. https://majadhondt.wordpress.com/2012/05/16/googles-9-lines/), even if it would be realistically near-impossible to detect if they 'stole' it.

streamofdigits• 5 years ago

exercise for the reader: the shortest program / keystrokes that would convert:

"democratize X" -> "capitalize on X hype"

NB: the challenge is that X can be any important sounding combination of words

m00dy• 5 years ago

Being an ex-googler still a thing nowadays ?

joaogui12• 5 years ago

I'll just say that the word democratize does not appear in Cohere's website

DebtDeflation2• 5 years ago

For example, Cohere is providing the NLP capability for Ada, a recent unicorn company in the chatbot space. Ada has experimented with the Cohere natural-language models to match customer chat requests with available support information. Rather than trying to anticipate all the possible wordings of a request, Cohere’s model tries to understand the intent behind it, Gomez says.
Cohere, he says, offers a platform containing a “full stack” of NLP functions, including sentiment classification, question answering, and text classification.

Google (Cloud Natural Language, Dialogflow), Microsoft (Azure LUIS, Bot Service), Amazon (Lex), and IBM (Watson Assistant, Watson Discovery) all offer Conversational AI and NLP APIs that do exactly what these guys are trying to do here. What is new or unique about them? I work in this space, and the Chatbot/Conversational AI market has been flooded with startups all doing the exact same thing for the last 3-4 years.

joaogui12• 5 years ago

ADA is the company doing chatbots/conversational AI, cohere is providing them with language models to do it, that's not the same thing as cohere being a company in the chatbot field

DebtDeflation2• 5 years ago

I understand that. However, the companies I listed are providing BOTH the raw NLP APIs as well as chatbot builder APIs.

Ex-Googlers raise $40M to democratize natural-language AI

Related Stories