iliane5

Just wanted to say thank you for your work and attention to detail, it's immensely valuable and we're all very grateful for it.

Watching an entirely generated video of someone painting is crazy.

I can't wait to play with this but I can't even imagine how expensive it must be. They're training in full resolution and can generate up to a minute of video.

Seeing how bad video generation was, I expected it would take a few more years to get to this but it seems like this is another case of "Add data & compute"(TM) where transformers prove once again they'll learn everything and be great at it

I think it's mostly the scale. Once you have a consistent user base and tons of GPUs, batching inference/training across your cluster allows you to process requests much faster and for a lower marginal cost.

What I was saying is that because you need to go out of your way to make sure it's tokenized properly, I wouldn't be surprised if there are enough non properly tokenized examples in the dataset.

If that was the case, it would make it difficult to generalize these concepts.

LLMs are not particularly good at arithmetic, counting syllables, or recognizing haikus

I suspect most of this is due to tokenization making it difficult to generalize these concepts.

There are some weird edge cases though, for example GPT-4 will almost always be able to add two 40 digits number but it is also almost always wrong when adding a 40 digit and 35 digit number.

AFAIK it's pretty standard practice not to expose the "raw" LLM directly to the user. You need a "sanity loop" where user input and the output of the LLM is checked by another LLM to actually enforce rules and mitigate prompt injections, etc.

100% agree.

However, seeing how excited Palantir is with their war assistant LLM , the US testing autonomous fighter jets a few months ago, etc. I think there's a decent chance that AI won't even have to break out of its constraints. It's pretty much guaranteed people are going to do the obviously dumb thing and give it capabilities it shouldn't have or is not equipped to deal with safely.

I don't think we need sentient AI for it to be autonomous. LLMs are powerful cognitive engines and weak knowledge engines. Cognition on its own does not allow them to be autonomous, but because they can use tools (APIs, etc.) they are able to have some degree of autonomy when given a task and can use basic logic to follow them through/correct their mistakes.

AutoGPTs and the likes are much overhyped (it's early tech experiments after all) and have not produced anything of value yet but having dabbled with autonomous agents, I definitely see a not so distant future when you can outsource valuable tasks to such systems.

Why is building what amounts to a calculator/spreadsheet/CAD program for language somehow a Rubicon that cannot be crossed?

We've already crossed it and I believe we should go full steam ahead, tech is cool and we should be doing cool things.

Did people freak out this much about computers replacing humans when they were shown to be good at math?

Too young but I'm sure they did freak out a little! Computers have changed the world and people have internalized computers as being much better/faster at math but exhibiting creativity, language proficiency and thinking is not something people thought computers were supposed to do.

There's no denying this is regulatory capture by OpenAI to secure their (gigantic) bag and that the "AI will kill us all" meme is not based in reality and plays on the fact that the majority of people do not understand LLMs.

I was simply explaining why I believe your perspective is not represented in the discussions in the media, etc. If these models were not getting incredibly good at mimicking intelligence, it would not be possible to play on people's fears of it.

The human brain works around a lot of limiting biological functions. The necessary architecture to fully mimic a human brain on a computer might not look anything like the actual human brain.

That said, there are 8B+ of us and counting so unless there is magic involved, I don't see why we couldn't do a "1:1" replica of it (maybe far) in the future.

I don't want every to know how to make a bomb.

This information is not created inside the LLMs, it's part of their training data. If someone is motivated enough, I'm sure they'd need no more than a few minutes of googling.

I do feel like this is more than a math formula

The sum is greater than the parts! It can just be a math formula and still produce amazing results. After all, our brains are just a neat arrangement of atoms :)

Why is it so hard to hear this perspective? Like, genuinely curious.

Because people have different definition of what intelligence is. Recreating the human brain in a computer would definitely be neat and interesting but you don't need that nor AGI to be revolutionary.

LLMs, as perfect Chinese Rooms, lack a mind or human intelligence but demonstrate increasingly sophisticated behavior. If they can perform tasks better than humans, does their lack of "understanding" and "thinking" matter?

The goal is to create a different form of intelligence, superior in ways that benefit us. Planes (or rockets!) don't "fly" like birds do but for our human needs, they are effectively much better at flying that birds ever could be.

Agreed, there is way too much hype about the actual capabilities of the LLaMa models. However, instruction tuning alone makes Alpaca much more usable than the the base model and to be fair even some versions of the "tiny" 7B can do small talk relatively well.

Using GPT to generate training data for fine-tuning seems to produce the best results, but even so, GPT4-x-Alpaca 30B is still clearly inferior to the real thing.

Distillation is interesting and it does seems to make the models adopt ChatGPT's style but I'm dubious that making LLMs generate entire datasets or copy/pasting ShareGPT is going to give you that great of a dataset. The whole point of RLHF is getting the human feedback to make the model better. OpenAI's dataset/RLHF work seems to be working wonders for them and will continue to give them a huge advantage (especially now that they're getting hundred of millions of conversations of people doing all sorts of things with ChatGPT)

I'm sure they're tweaking lots of things under the hood, especially now that they have 100M+ users. It could be bigger (30B?, maybe 65B) as coming down from 175B gives quite a lot of room, but the cognitive drop from Davinci gives away that's it's much smaller.

People fine-tuning LLaMa models on arguably not that much/not the highest quality data are already seeing pretty good improvements over the base LLaMa, even at "small" sizes (7B/13B). I assume OpenAI has access to much higher quality data to fine-tune with and in much higher quantity too.

GPT-3.5 is much worse at "complex" cognitive tasks than Davinci (175B), which seem to indicate that it's a smaller model. It's also much faster than Davinci and costs the same as Curie via the API.

It’s not only 10x cheaper, it’s also way faster at inference and not as smart as Davinci. IMO the only logical answer is that the model is just smaller.

I bet they’re not saying how big of a model GPT-4 is because it’s actually much smaller we would expect.

ChatGPT is IMO a heavily fine-tuned Curie sized model (same price via API + less cognitive capacity than even text davinci-003) so it would make sense that a heavily fine-tuned Davinci sized model would yield similar results to GPT-4.

I think as soon as text2video gets really good (like midjourney level), there’s gonna be so much AI generated content that unless it’s all extremely good, human made content will be something people search specifically for.

As for curation, I think the success of TikTok proves that you don’t need that much data to pretty preceding pinpoint what someone wants to watch (or what will get them to spend the most time on the app at least).

LLMs architected and trained the way they are now can never approach human reasoning capability

Not sure if you’ve played with GPT-4 but honestly it’s getting there. If you take the bar exam, ChatGPT was in the bottom 10% of participants, GPT-4 is in the top 90%.

It obviously isn’t the ultimate test of reasoning/intelligence but I think we would agree that a human who’s in the top 90% is likely to be pretty smart.

Of course LLMs aren’t the final word in AI development

Couldn’t agree more. AGI will come from plugging a few of these systems together.

Absolutely. What’s fascinating is that they’re getting such good understanding of many things through just text. Multimodal models that can process text, images, sounds, video, etc. are gonna be very interesting for that very reason

There's no denying LLMs are anything but sentient however is sentience really needed for intelligence? I feel like if we can have machines that are X% smarter than a human could ever get for any given task, it'd be a much better outcome for us if they were not sentient.

But it's not better than almighty human intelligence, it _is_ human intelligence, because it was trained on a mass of some of the best human intelligence in all recorded history

Sure, I was saying "better" in the sense that if for X task, it can do better than Y% of humans.

since we hand-fed it the answers, it falls a little flat for me

We didn't really hand-fed it any answers though did we? If you put a human in a white box all its life, with access to the entire dataset on a screen but no social interaction, nothing to see aside from the text, nothing to hear, nothing to feel, nothing to taste, etc, it'd be very impressed if they were then able to create answers that seem to display such thoughtful and complex understanding of the world.

Well it's like birds and airplanes. Do airplanes "fly" in the same sense that birds do? Of course not, birds flap their wings and airplanes need to be built, fueled and flown by humans. You could argue that the way birds fly is "more natural" or superior in some ways but I've yet to see a bird fly Mach 3.

If you replace the analogy with humans and LLMs, LLMs won't ever reason or understand things in the same way we do, but if/when their output gets much smarter than us across the board, will it really matter?

What I find really entertaining is the "just predicting the next token" argument.

If just predicting the next token can produce similar or better results than the almighty human intelligence on some tasks, then maybe there's a bit of hubris in how smart we think we actually are.

SVG also supports <script> tags, which is even wilder

Apparently it’s for analytics

Spam is spam whether made by a human or an AI.

Good, valuable content is good whether made by a human or an AI.

Bad or uninteresting content is spam if it’s being pumped out by an AI, but it’s a learning experience/personal expression if it’s made by a human.

The difference is intent.

AI content detection is a losing battle imo. Models will become better and people will use the detectors to rephrase the text so it doesn’t get detected.

I suppose if the text is good and adds value and is factually correct then it does not matter if it's a human or an AI assistant who wrote it.

The real issue is the bad content. It’s perfectly acceptable for a human to publish a bad article/blog post but when language models do it it’s more or less spam.