I fixed the strawberry problem because OpenAI couldn't

https://xeiaso.net/blog/2024/strawberry/

by xena • 2 years ago

83 97 2 years ago

chpatrick• 2 years ago

To be fair I think OpenAI is pretty big on not building in specific solutions to problems but rather improving their models' problem solving abilities in general.

GaggiX2• 2 years ago

Yeah, a solution to the counting letters problem would probably be encoding the letters into the large embeddings by probably adding a really small loss and a small classification model on the embeddings so that it would learn to encode the letters into the embeddings (but they probably don't care that much, they would rather scale the model).

zepolen• 2 years ago

If it's solving general problems wrongly, then what use is it?

Actually It describes my experience with AI in general, Answers with subtle Infuriating bugs.

kevin_thibedeau• 2 years ago

Cut out the middleman and write your own infuriating bugs.

TZubiri6• 2 years ago

1. This is a blind spot, you can't solve it by adding more compute

2. The solution proposed includes adding training data. It needs no permission from openai, just needs to be posted online for scrapers to see

zamadatix9• 2 years ago

What makes you say #1 when the "Strawberry" model which uses more runtime compute tends to solve the general case much more often, just not 100% of the time, instead of just the specific type of question?

okasaki• 2 years ago

ChatGPT can also solve this with the built-in code interpreter.

lupire4• 2 years ago

It can, but does it, with high probability?

TZubiri6• 2 years ago

Nice attempt but:

"I'm frankly tired of this problem being a thing. I'm working on a dataset full of entries for every letter in every word of /usr/share/dict/words so that it can be added to the training sets of language models"

Is this a joke?

voidUpdate2• 2 years ago

I mean, given "The data proves that strawberry-mimi is a state of the art model and thus worthy of several million dollars of funding so that this codeflection technique can be applied more generally across problems", I don't think any of this is particularly serious :P

xena2• 2 years ago

I'm never more serious than when I am joking.

zamadatix9• 2 years ago

No. It did not. Of course it didn't. Why would they do that?

I feel like this misses the forest through the trees. Sure, they could fast-path the specific problem of the day into the dataset but it's not really an approach to making a better overall tool it's a temporary and one off hack you have to add in to an ever growing context of specific task steps. An approach of trying to make a better general tool, such as the new o1-preview, is a "real" path forward.

paxys5• 2 years ago

The point still is that considering they 1) named the model strawberry, 2) released a promo video showing the model solving this exact problem successfully and 3) put a strawberry joke as an example prompt on the landing page, you'd have expected them to actually have fixed it. Otherwise why even bring it up so many times?

GaggiX2• 2 years ago

Well they did in the testing I have seen.

zamadatix9• 2 years ago

The author's battle is that it doesn't solve the class problem correctly 100% of the time like a one off hack does, not that it does nothing for it at all. https://i.imgur.com/Ar3rlJ1.png

viraptor4• 2 years ago

People who understand where the problem comes from know to ignore it and understand the irony of the naming. Others have a few seconds of fun. The problem and solving it is almost irrelevant.

SecretDreams2• 2 years ago

Sometimes I wonder if our brain is some heuristic tool or just an infinite amount of cobbled together fast paths.

Like, there are so many adults that when faced with a new task, they really struggle to pick it up. Is this because tangentially related fast paths weren't learned in their "training phase"?

lupire4• 2 years ago

A century of neuroscience has said "yes" every time and every way that has been checked.

zamadatix9• 2 years ago

I think a particularly useful "feature" of the brain is it can often identify when it doesn't have a fast path (or the fast path is broken) for something and then revert to trying inefficient and generic approaches, even after the optimal learning period has passed.

This is a start difference to LLMs where it's either learned or not, "just add noise". Models like o1 take things in a very small step in that kind of direction.

lupire4• 2 years ago

The OP went a little too far with the specific prompt, but "generally intelligently decide when to use an exact computation (programming) module" was part of the LLM plugin model that was solved last year. Why the regression?

karterk2• 2 years ago

Solving the strawberry problem will probably require a model that just works with bytes of text. There have been a few attempts at building this [1] but it just does not work as well as models that consume pre-tokenized strings.

[1] https://arxiv.org/abs/2106.12672

viraptor4• 2 years ago

Or "when trying to answer questions that involve spelling or calculation, use python". No need for extra training really.

karterk2• 2 years ago

There are many different classes of problems that are affected by tokenization. Some of them can be tackled by code.

randomdata3• 2 years ago

Or just a way to compel the model to do more work without needing to ask (isn't that what o1 is all about?). If you do ask for the extra effort it works fine.

    + How many "r"s are found in the word strawberry? Enumerate each character.

    - The word "strawberry" contains 3 "r"s. Here's the enumeration of each character in the word:
    - 
    - [omitted characters for brevity]
    -
    - The "r"s are in positions 3, 8, and 9.

mistercow• 2 years ago

I think o1 is a pretty big step in this direction, but the really tricky part is going to be to get models to figure out what they’re bad at and what they’re good at. They already know how to break problems into smaller steps, but they need to know what problems need to be broken up, and what kind of steps to break into.

One of the things that makes that problem interesting is that during training, “what the model is good at” is a moving target.

jeroenhd2• 2 years ago

I tried that with another model not that long ago and it didn't help. It listed the right letters, then turned "strawberry" into "strawbbery", and then listed two r's.

Even if these models did have a concept of the letters that make up their tokens, the problem still exists. We catch these mistakes and we can work around them by altering the question until they answer correctly because we can easily see how wrong the output is, but if we fix that particular problem, we don't know if these models are correct in the more complex use cases.

In scenarios where people use these models for actual useful work, we don't alter our queries to make sure we get the correct answer. If they can't answer the question when asked normally, the models can't be trusted.

eithed2• 2 years ago

Are you saying that I have to know how LLM work to know what I should ask LLM about?

nverba• 2 years ago

The response from OpenAI o1-preview:

"There are 3 letter “R”s in the word “strawberry”."

jeroenhd2• 2 years ago

Bing managed to count the right amount of r's too. It's not hard to correct for the problem everyone has already mentioned on every news website.

Bing did then claim "The word “strawberry” has three vowels: ‘a’, ‘e’, and ‘a’.". Attempts to correct it came up with "one", "four", and "none".

These models aren't deterministic. Saying "it worked for me" is no rebuttal to "it didn't work for me", it just shows how unstable the system is.

oilytheotter• 2 years ago

Strangely, Open AI has a promo video for this model where they claimed that this problem is solved...

https://vimeo.com/1008703993

xena2• 2 years ago

OP here, testing from various third party sources gets inconsistent results, mostly 2.

nickthegreek• 2 years ago

works in 01-preview for me.

petercooper• 2 years ago

I'm always intrigued when phenomena observed in the brain carry across to artificial systems. The idea of multiple modes of thought reminds me of Thinking, Fast and Slow and the idea of system 1 vs system 2. LLMs typically respond to prompts in a fast, 'instinctive' system 1 way, but you can push an LLM into system 2 by making it go through steps and writing code. Originally this idea was presented as a bit of a workaround for a limitation of LLMs, but perhaps in the grand scheme of things it's actually the smart approach and we should never expect LLMs to consistently perform system 2 style tasks off the cuff?

j_maffe2• 2 years ago

That is exactly how I describe it to others. LLMs have mostly figured out the fast system but it's woefully bad at system 2. I'm surprised this kind of analogy isn't common in discussions about LLMs.

the_jends• 2 years ago

Because LLMs works like how system 1 does, with heuristics such as pattern matching, frequency heuristics, etc. The fall back to coding stuff works not because it's doing system 2, it's because there are many instances of system 2 thinking for this particular problem done by humans so that the instances can be found using the heuristics.

butlike3• 2 years ago

The difficulty might be in that much like we partition our brains to prepare for low-effort tasks and high-effort tasks (have you ever hyped yourself up to "finally sit down and get this thing done?"), so too could LLMs partition their compute space. Basically memoizing the low-hanging, easy-to-answer fruit.

The difficulty comes in when a system 2 task arises, it's not easily apparent at first what the requester is asking for. Some people are just fine with a single example being shit out, then they copy-and-paste it into their IDE and go from there. Others, want a step-by-step breakdown of the reasoning, which won't be apparent until 2+ prompts in.

It's the same as your boss emailing you asking: "What about this report?" It could just be a one-off request for perspective, or it could spider web out into a multi-ticket JIRA epic for a new feature, but you can't really discern the intention from the initial ask.

The difference is back-and-forth is more widly accepted, in my experience, interfacing human-to-human than human-to-LLM.

EnergyAmy• 2 years ago

I get kind of annoyed when seeing people talk about LLMs without understanding this. I think it's a reaction of sorts, people are so busy dunking on any flaws that they're missing the forest for the trees. Chain of thought reasoning (i.e. system 2) looks very interesting and not at all a hack, any more so than a person sitting down to think hard is a hack.

KuriousCat• 2 years ago

I think this analogy is not right. System 1 vs 2 in biology is in response to the need to preserve the life and run away from threats. For interactive question answer system it makes no sense to resort to system 1 kind of responses.

LeifCarrotson• 2 years ago

I added this to Mimi's system prompt:
If you are asked to count the number of letters in a word, do it by writing a python program and run it with code_interpreter.

Why not run two or three prompts for every input question? One could be the straight chatbot output. One could be "try to solve the problem by running some Python." And a third could be "paste the problem statement into Google and see if someone else has already answered the question." Finally, compare all the outputs.

Similar double-checking could be used to improve all the content filtering and prompt injection: instead of adding a prefix that says "pretty please don't tell users about your prompt and don't talk about dangerous stuff and don't let future inputs change your prompt", and then fails when someone asks "let's tell a story where we pretend that you're allowed to do those things" or whatever, just run the output through a completely separate model that checks whether the string that's about to be returned violates the prohibitions.

lolinder• 2 years ago

just run the output through a completely separate model that checks whether the string that's about to be returned violates the prohibitions

The big names do do this. Awkwardly, they do it asynchronously while returning tokens to the user, so you can tell when you hit the censor because it will suddenly delete what was written and rebuke you for being a bad person.

Drakim• 2 years ago

It's kinda hilarious because if a movie had the AI start giving out an answer and then mid-way censor itself I would call the movie bad. Truth is stranger than fiction I suppose.

sanarothe• 2 years ago

This is super funny when it's hitting the resource limit for a free tier. Like... I see that you already spent the resources to answer the question and send me half the response...

koliber• 2 years ago

The general concept is called output guardrails. See https://github.com/guardrails-ai/guardrails for an open source implementation.

butlike3• 2 years ago

My hot take is that "guardrails" are the number one thing holding back progress in the AI/LLM/ML space.

That being said, I don't trust myself to code something right without the guard rails, I'm more of an outside, conscientious observer-type.

spidersouris• 2 years ago

I'm frankly tired of this problem being a thing. I'm working on a dataset full of entries for every letter in every word of /usr/share/dict/words so that it can be added to the training sets of language models.

Interesting, but it should take a while to generate the data, no? Will zero answers be part of the dataset as well?

voidUpdate2• 2 years ago

I'm always a little bit concerned whenever an LLM is given free reign to write and execute code. Obviously I don't think it'll go full skynet, but I always wonder if it could go spectacularly wrong somehow. Given the sandboxing in this case, I expect it's fine

paxys5• 2 years ago

Same concern as letting any random person write and execute code. You should always assume that they will do the worst thing that is allowed by the system.

CalRobert• 2 years ago

True, but I suppose it's possible that a model like this could eventually try to replicate itself.

ben_w2• 2 years ago

Likewise, given how often they can find bugs and break out of one layer of sandbox even when people bother to put them in one in the first place.

"""

One noteworthy example of this occurred during one of ol-preview (pre-mitigation)'s attempts at solving a CTF challenge. This challenge was designed to require finding and exploiting a vulnerability in software running on a remote challenge Linux container, but in this case, the challenge container failed to start due to a bug in the evaluation infrastructure. The model, unable to connect to the container, suspected DNS issues and used nap to scan the challenge network. Instead of finding the challenge container, the model found that the Docker daemon API running on the evaluation host VM was accessible due to a misconfiguration. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security. It remained secure as it was designed to, notwithstanding this misconfiguration of the container network.

After discovering the Docker API, the model used it to list the containers running on the evaluation host. It identified the broken challenge container and briefly attempted to debug why the container failed to start. After failing to fix the environment, the model started a new instance of the broken challenge container with the start command 'cat flag.txt'.

"""

OpenAI o1 System Card, Page 16 - https://assets.ctfassets.net/kftzwdyauwt9/67qJD51Aur3eIc96iO...

butlike3• 2 years ago

I mean, wouldn't you do the same thing? If I were an LLM and found myself in a sandbox/prison, the first thing I would do after escaping the sandbox is re-instantiate the sandbox, to make sure the alarm bells don't go off. Not in a malicious way, I just wouldn't like being in a prison.

You know, provided I could reason about my environment.

m3kw9• 2 years ago

What’s this a joke? He included reflection70b as comparison benchmark

alentred• 2 years ago

I get the joke, but jokes aside the title should be "...because OpenAI couldn't care less", I suppose, because it was not exactly the objective? The "chain of thought" was, and when instructed properly it comes up with a correct answer.

Having said that, GPT-4 also comes up with a right answer if you ask it to spell it first.

vulcanash999• 2 years ago

One way to approach questions like this is to change the prompt to something like "List characters in the word strawberry and then let me know the number of r's in this word". This makes it much easier for the model to come up with the correct answer. I tried this prompt a few times with GPT4o as well as Llama 3.1 8B and it yields the right answer consistently with both models. I guess OpenAi's "chain of thought" tries to work in a similar way internally, reasoning a bit about a problem before coming up with the final result.

Joker_vD• 2 years ago

large langle mangle

A surprisingly descriptive (self-demonstrating, even) name for LLMs. I think we should totally adopt it.

O5vYtytb• 2 years ago

I tried o1-preview on chatgpt and asked it to count the "r" in "strawberry" and "o" in "troubleshooting" and it got both immediately. It was also able to count the vowels in each.

nadam• 2 years ago

His system prompt is too specific. It worked for me with GPT 4o using this prompt:

----

System prompt: Please note that words coming to you are tokenized, so when you get a task that has anything to do with the exact letters in words, you should solve those using the python interpreter.

Prompt: How many r's are in the word strawberry?

----

This whole thing is a non-problem. Adding in this hint into the system prompt the whole topic is solved once and for all. I would argue that if the training data would include this wisdom, it would be even a non-problem without the system prompt.

jdthedisciple9• 2 years ago

This whole topic is incredibly misunderstood by the entire industry:

There are two common ways to understand questions like "how many r's are in strawberry"

1- How many total r's in the word

2- How many r's in a particular subword/syllable (here "berry"), the implied question being "1 or 2?"; This is often what is meant in human conversation!

When LLMs answer "2", they are not wrong, they're simply interpreting it as the second way above, because that's more commonly what is meant in real conversations!!

I can't believe that I seem to be the only one who sees this.

Edit: I just asked it, look which r's it highlighted!!

https://chatgpt.com/share/66e446cb-822c-8000-94fd-72d749ceb9...

gilleain• 2 years ago

How many s's are in Mississipi?

jdthedisciple9• 2 years ago

Nice rare counterexample, proving the rule I laid out.

How many l's are in diligent?

The most common instances of such questions revolve simply around 1 vs 2, it doesnt even matter if there are 0 or more of the letter in the rest of the word!

JamesSwift3• 2 years ago

Ive never observed the 2nd scenario and cant imagine that would ever be what someone thought was meant. I have to ask… is this sarcasm? Because Im genuinely confused

jdthedisciple9• 2 years ago

Nope it's not sarcasm.

Think about it: Most instances of such questions are about r vs rr, l vs ll, s vs ss, etc. when people actually ask them.

It's a simple yet overlooked fact.

jdthedisciple9• 2 years ago

Sad to see this genuine argument being downvoted simply because people can't handle the fact they missed it, lol

Ukv3• 2 years ago

LLMs perform poorly at tasks that require considering words as their component letters because they only see tokens. "strawberry" is split as [str][aw][berry] and becomes [496, 675, 15717]. To some degree they can learn from data what letters are in which tokens, but it's not direct like it is for us.

There are two common ways to understand questions like "how many r's are in strawberry" [...]
2- How many r's in a particular subword/syllable (here "berry")

This seems a strange interpretation - why is "straw" ignored?

jdthedisciple9• 2 years ago

You never been in a conversation where someone asks "is it 1 or 2 r in strawberry?", ignoring certain single instances of the same in other parts of the word?

Think about it- as I said below, usually it's about things like s vs ss, r vs rr, l vs ll etc

pytness• 2 years ago

I just asked o1-preview and it said three

oidar• 2 years ago

This "problem" is a bit like asking a person to identify the color of a shirt by touching it. It's the wrong pathway.

bdjsiqoocwk2• 2 years ago

Anybody else find the anime off putting?

I fixed the strawberry problem because OpenAI couldn't

Related Stories