HN user
It's already happening.
“ChatGPT, a version of OpenAI’s GPT-3.5 model… gained more than 100m users in its first two months, and is now estimated to produce a volume of text every 14 days that is equivalent to all the printed works of humanity.”
— Dr Thompson, Feb/2023, cited in report by the National Bureau of Economic Research (Scholes, Bernanke, MIT)
https://www.nber.org/system/files/working_papers/w30957/w309...
Just for clarity, this is Azure[1].
[1] https://venturebeat.com/ai/openai-is-reducing-the-price-of-t...
If you're talking about Google Bard, they were very clear in the LaMDA 2 paper that they only used public sources.
"...from public dialog data and other public web documents..."
LaMDA 2 paper: https://arxiv.org/abs/2201.08239
My overview of Google Bard including dataset: https://lifearchitect.ai/bard/
My overview of Google PaLM and Pathways family including dataset: https://lifearchitect.ai/pathways/
Compare with other models including the use of DeepMind's MassiveWeb/MassiveText and EleutherAI's Pile dataset: https://lifearchitect.ai/whats-in-my-ai/
Actually, it's 'BookCorpus'. OpenAI spelt it wrong in their GPT-1 paper.
It has also been analyzed here and here:
Ilya used that term a lot, and continues to do so.
https://www.newyorker.com/magazine/2019/10/14/can-a-machine-...
https://twitter.com/ilyasut/status/1486728489319362571?lang=...
Full text: https://lifearchitect.ai/bing-chat/
ChatGPT is probably outputting at least 110x the equivalent volume of Tweets by human Twitter users every day.... and the equivalent of the entire printed works of humanity every 14 days (Jan/2023)
The ChatGPT Prompt Book: https://docs.google.com/presentation/d/17b_ocq-GL5lhV_bYSShz...
lol!
Excellent candidate for https://www.reddit.com/r/dontyouknowwhoiam/
Video demo: https://youtu.be/B7Mg8Hbcc0w
More info on Claude's principles/Constitution: https://lifearchitect.ai/anthropic/
Video walkthrough and interview with creator James Weaver: https://youtu.be/wYGbY811oMo
The human feedback fine-tuning concept was applied following strict policies and rules. The rules chosen by OpenAI would be very similar to those applied by DeepMind for the Sparrow dialogue model (Sep/2022), which is a fine-tuned version of DeepMind’s Chinchilla model.
The rules used for DeepMind Sparrow were selected by researchers from DeepMind (Alphabet), California Institute of Technology, University of Toronto, and University College Dublin.
This is an older paper, and demo'd here with GPT-3 instead of Chinchilla.
Video of DeepMind Dramatron (Chinchilla version):
Here's a full list of the big ones:
https://lifearchitect.ai/models/
https://lifearchitect.ai/timeline/
There are another 97,000 Transformer models on HF:
Maybe back in June 2022.
Since then, around 70,000 new Transformer models have been released via HF.
There are several models much larger than YaLM, including BigScience BLOOM 176B and Tsinghua GLM-130B.
In 2020, GPT-3 175B was already outperforming humans on SAT questions by ~20%. There are far more formal benchmarks for models now.
A list of GPT-3 and Codex clients was announced by Microsoft’s Satya Nadella and OpenAI’s Sam Altman in October 2022:
Accenture, AI Busters, Alert Innovation, American Fidelity, Aon, Arvato Bertelsmann, Autodesk, Avanade, BMW, Carmax, Cipio.ai, Clevertar, Cognizant, Databook, Downer, EY, Farmlands Cooperative, Genie AI, Elsevier Health, HSBC, IFAD, Inpris, Inward, Kept, Klaviyo, Morgan Stanley, National Nederlanden, PwC, Shell, Snelstart, Sogeti, Soul Machines, Strabag, Trelent, Wipro, WordLift, Zammo.ai
Very high-level...
Models: https://lifearchitect.ai/models/
Timeline: https://lifearchitect.ai/timeline/
Text-to-image: https://lifearchitect.ai/art/
SymphonyNet: https://youtu.be/m4tT5fx_ih8
Dude is 95 years old, and one of the most effective public speakers I have ever seen.
“[Cisco executive] Charles Giancarlo fielded a call directly from Steve Jobs. “Steve called in and said that he wanted it,” Giancarlo recalled. “He didn’t offer us anything for it. It was just like a promise he’d be our best friend. And we said, ‘No, we’re planning on using it.’ “Shortly after that, Apple’s legal department called to say they thought Cisco had “abandoned the brand,” meaning that in Apple’s legal opinion Cisco hadn’t adequately defended its intellectual property rights by promoting the name. To Apple’s way of thinking this meant the name iPhone was available for Apple’s use. Giancarlo, who subsequently joined the prominent Silicon Valley private-equity firm Silver Lake partners, said Cisco threatened litigation before the launch. Then, the day after Apple announced its iPhone, Cisco filed suit.
The negotiation displayed some classic Steve Jobs negotiating tactics. Giancarlo said Jobs called him at home at dinnertime on Valentine’s Day, as the two sides were haggling. Jobs talked for a while, Giancarlo related. “And then he said to me, ‘Can you get email at home?'” Giancarlo was taken aback. This was 2007, after all, when broadband Internet was ubiquitous in homes in the US, let alone that of a Silicon Valley executive who had worked for years on advanced Internet technology. “And he’s asking me if I’m able to get email at home. You know he’s just trying to press my buttons—in the nicest possible way.” Cisco gave up the fight shortly after that. The two sides reached a vague agreement to cooperate on areas of mutual interest.”
https://www.idownloadblog.com/2012/01/27/apple-cisco-iphone-...
I also noticed the writing was particularly poor. And not just the technical detail... Everything from grammar to general syntax needs tweaking for ease-of-reading.
That was a wild ride...