LLMs and the Harry Potter problem

https://www.pyqai.com/blog/llms-and-the-harry-potter-problem

by araghuvanshi • 2 years ago

65 61 2 years ago

iAkashPaul• 2 years ago

Seems like using RAFT(https://techcommunity.microsoft.com/t5/ai-ai-platform-blog/r...) to generate the glossary+KGs for a given doc should give better results for cross-doc or intra doc answers instead of just relying on vDB chunks alone.

araghuvanshi16• 2 years ago

Good share, thank you! Yeah I think Contextual AI has also been doing some interesting work in this area. Glossary is definitely interesting and an area we're looking into. Curious to see what work is being done with building knowledge graphs, that's another area where we've seen positive results.

StephenSmith• 2 years ago

"If the only tool you have is a hammer, you tend to see every problem as a nail." -Abraham Maslow

madrox• 2 years ago

I, too, could not read a chapter of Harry Potter and then tell you how many times a word was used. This isn't what my brain (and by extension LLMs) is good at. However, if you told me ahead of time that was my goal for reading a chapter, I'd approach reading differently. I might have a scratch pad for tallying. Or I might just do a word find on a document. I'd design a framework to solve the problem.

"The Harry Potter Problem" has the feel of a strawman. LLMs are not universal problem solvers. You still have to break down tasks or give it a framework for working things through. If you ask an LLM to produce a code snippet for word counting, it will do great. Maybe that isn't as sexy, but what are you really trying to achieve?

jackpirate• 2 years ago

They very clearly explain why this matters in the "Why should I care?" section. Partially quoting them:

Harry Potter is an innocent example, but this problem is far more costly when it comes to higher value use-cases. For example, we analyze insurance policies. They’re 70-120 pages long, very dense and expect the reader to create logical links between information spread across pages (say, a sentence each on pages 5 and 95). So, answering a question like “what is my fire damage coverage?” means you have to read: Page 2 (the premium), Page 3 (the deductible and limit), Page 78 (the fire damage exclusions), Page 94 (the legal definition of “fire damage”).

It's not at all obvious how you could write code to do that for you. Solving the "Harry Potter Problem" as stated seems like a natural prerequisite for doing this much more high stakes (and harder to benchmark) task, even if there are "better" ways of solving the Harry Potter problem.

causal• 2 years ago

Word counting and summarizing key information are wildly different problems though

famouswaffles4• 2 years ago

Not really.

LLMs can't count well. This is in large part a tokenization issue. Doesn't mean they couldn't answer all those kind of questions. Maybe the current state of the art can't. But you won't find out by asking it to count.

lolinder5• 2 years ago

Solving the "Harry Potter Problem" as stated seems like a natural prerequisite for doing this much more high stakes (and harder to benchmark) task

Not really. The "Harry Potter Problem" as formulated is asking an LLM to solve a problem that they are architecturally unsuited for. They do poorly at counting and similar algorithms tasks no matter the size of the context provided. The correct approach to allowing an AI agent to solve a problem like this one would be (as OP indicates) to have it recognize that this is an algorithmic challenge that it needs to write code to solve, then have it write the code and execute it.

Asking specific questions about your insurance policy is a qualitatively different type of problem that algorithms are bad at, but it's the kind of problem that LLMs are already very good at in smaller context windows. Making progress on that type of problem requires only extending a model's capabilities to use the context, not simultaneously building out a framework for solving algorithmic problems.

So if anything it's the reverse: solving the insurance problem would be a prerequisite to solving the Harry Potter Problem.

araghuvanshi16• 2 years ago

Some counterarguments: 1. If an AI company promises that their LLM has a million token context window, but in practice it only pays attention to the first and last 30k tokens, and then hallucinates, that is a bad practice. And prompt construction does not help here - the issue is with the fundamentals of how LLMs actually work. Proof: https://arxiv.org/abs/2307.03172 2. Regarding writing the code snippet: as I described in my post, the main issue is that the model does not understand the relationships between information in the long document. So yes, it can write a script that counts the number of times the word "wizard" appears, but if I gave it a legal case of similar length, how would it write a script that extracts all of the core arguments that live across tens of pages?

doug_durham3• 2 years ago

I'd do it like a human would. If a human was reading the legal case they would have a notepad with them where they would note locations and summaries of key arguments, page by page. I'd code the LLM to look for something that looks like a core argument on each page (or other meaningful chunk of text) and then have it give a summary if one occurs. I may need to do some few shot prompting to give it understanding of what to look for. If you are looking for reliable structured output you need to formulate your approach to be more algorithmic and use the LLM for it's ability to work with chunks of text.

kemiller• 2 years ago

Yeah. Humans can't count more than ~5 things intuitively; we have to run the "counting algorithm" in our heads. We just learn it so early in life that we don't really think of it as an algorithm. Not surprising at all that LLMs have the same limitation, but fortunately computers are extremely good at running algorithms once instructed to do so.

jmkni• 2 years ago

But you could write code to do exactly that easily though

So surely it’s not about LLMs being able to do that, but being smart enough to understand “hey this is something I could write a Python script to do” and be able to write the script, feed the Harry Potter chapter into it, run the script and parse the results (in the same way a human would do)?

lolinder5• 2 years ago

Yes, and that's a completely different kind of problem than extending a model's ability to use its context window effectively.

If you solve the problem that way you don't even really need the Harry Potter chapter in the context at all, you could put it as an external document that the agent executes code against. This makes it qualitatively a different problem than the insurance policy questions that the article moves on to.

iLoveOncall• 2 years ago

I, too, could not read a chapter of Harry Potter and then tell you how many times a word was used. This isn't what my brain (and by extension LLMs) is good at. However, if you told me ahead of time that was my goal for reading a chapter, I'd approach reading differently. I might have a scratch pad for tallying. Or I might just to a word find on a document. I'd design a framework to solve the problem,

What is the relevance in what a human would do? This is not a human and does not work like a human.

I would expect any piece of software that allows me to input a text and ask it to count occurences of words to do so accurately.

doug_durham3• 2 years ago

Why would you expect that? You have no basis to have that expectation. The product isn't being presented as something that can do that.

nicklecompte• 2 years ago

You absolutely could count the number of times "wizard" was used if you had the book in front of you. Similarly the LLM does have the chapter available "to look at." Documents pasted in the context window aren't ethereal.

This explanation/excuse doesn't hold water.

lolinder5• 2 years ago

Part of the confusion here is that some people (apparently including you) use the word "LLM" to refer to the entire system that's built up around the language model itself, while others (like OP) are specifically referring to the large language model.

The large language model's context window absolutely is ephemeral. By the time inference is begun all you have is a giant vector that represents the context to date. This means that the model itself does not have the text available to look at, it only has the encoded "memory" of that text.

OP is simply saying that the underlying model is unsuitable for solving problems like this directly, so it makes a bad example for how models don't use their context effectively. A production grade AI agent should be able to solve problems like this, but it will likely do that through external scaffolding, not through improvements to the model itself, whereas improvements to the context window will probably need to occur at the model level.

trash_cat• 2 years ago

This is a strange example. A LLM would write a script for you to find the amount of "Wizard" being mentioned. That is what it always doing when it comes to numbers because it knows that it is its weakness.

Edit: Going from counting numbers to finding relevant information at different pages given a task isn't a valid analogy.

Agents are mentioned but not multi-agent architectures [0], where you could have an agent responsible for insurance policies, legal definitions, and/or a bot that is responsible for the big picture in question. They would go back and forth, being expert at their field (or task) and come to a conclusion after some iterations of API calls.

Missed opportunity.

[0] https://microsoft.github.io/autogen/docs/Use-Cases/agent_cha...

araghuvanshi16• 2 years ago

Please see my comment below, and the "Why should I care" section of the post. Yes you can count the number of times the word "wizard" is mentioned, but for tasks that aren't quite as cut-and-dry (say, listing out all of the core arguments of a 100-page legal case), you cannot just write a Python script.

The agentic approach falls apart because again, a self-querying mechanism or a multi-agent framework still needs to know where in the document to look for each subset of information. That's why I argue that you need an ontology. And at that point, agents are moot. A small 7b model with a simple prompt suffices, without any of the unreliability of agents. I suggest trying agents on an actually serious document, the problems are pretty evident. That said, I do hope that they get there one day because it will be cool.

famouswaffles4• 2 years ago

LLMs see tokens not words and counting is a problem for them, high context or no.

Maybe the current state of the art LLM can't solve the kind of high value long context problems you have in mind but what I can tell you though is that you won't find that out by asking it to count.

polotics2• 2 years ago

One of the big issues in this space seems yo be the wishful thinking. Have you tried to use Autogen to solve any of the hard questions mentioned in the article? I had a good go at a specific domain, witb a corpus of only a dozen thousand pages, and I agree with the article: your agents need to build a solid ontology, and if you found a way to get Autogen to do that, then please please tell us.

educasean• 2 years ago

Not only is "Harry Potter problem" a misnomer, the "shortcomings" of LLM being investigated here doesn't feel novel. We know counting is a big weakness of LLMs already, so why muddle the waters if you wanted to drill down on issues related to large context recall?

Perhaps my biggest gripe: if you're going to lure readers in with an interesting name like the "Harry Potter problem", it better be either technically interesting or entertaining.

araghuvanshi16• 2 years ago

Do most readers know that if you give a so-called million token context model that many tokens, it'll actually stop paying attention after the first ~30k tokens? And that if they were to try to use this product for anything serious, they would encounter hallucinations and incompleteness that could have material implications?

Not everything needs to be entertaining to be useful.

lolinder5• 2 years ago

The point is that this isn't even really useful because it's not a minimum reproduction of the problem they're actually interested in.

LLMs are bad at counting no matter what size of context is provided. If you're going to formulate a thought experiment to illustrate how an LLM stops paying attention well before the context limit, it should be an example that LLMs are known to be good at in smaller context sizes. Otherwise you might be entertaining but you're also misleading.

thomastjeffery2• 2 years ago

It blows my mind how absurd the narrative around LLMs has gotten. It's incredible that this subject has been talked in circles so far that trivial realities like this aren't immediately obvious to everyone.

LLMs do not count. When you ask an LLM to count (which is something it does not do), in the end no counting has happened. No shit, Sherlock.

polotics2• 2 years ago

So what is your point?

Claims of context-window sizes increases do imply that information can be extracted from such windows. So this is not trivial.

The counting example is a simple case, the "how much is this insurance policy going to cover this damage" is also a calculation.

thomastjeffery2• 2 years ago

But the LLM never calculates anything! Why in the world would I ever expect a calculated result?

An LLM doesn't even answer a question, either: it continues a prompt. That continuation looks like an answer to you and me; but to the LLM, it contains nothing more than the most likely tokens.

I keep hearing this story that an LLM is capable of objectivity, and that it's just relatively bad at it. The LLM literally never does objectivity. It can't be bad at a thing it never does.

Every time someone calls this sort of interaction a "limitation", they are only obfuscating the narrative with a useless anthropomorphization. The LLM is not a person. It is not a mind. It does not perform objective thought. It is a statistical model that provides the most likely text. No more, no less.

amiantos2• 2 years ago

I think some commenters are missing the point of this post by pointing out an LLM is the wrong tool for the Harry Potter task, because that's literally the point of the post, that people are actively trying to use LLMs in this way, right now. A lot of hucksters and former/current sales people, specifically.

The reason long context windows are advertised is because there are an awful lot of people out there trying to make money replacing customer service agents with LLM-powered chatbots. In order to power them naively (which is the only way sales people who have never built software before know how), you need to feed them a context window full of all your industry/product specific knowledge and then hope that the LLM answers the same way.

But, they don't, and they can't, so you have to spend a lot of time trying to figure out how to tie the LLM down so it responds exactly the way you want, which sort of ruins all the hype and mystery for common people around LLMs. It's been sold as a miracle that can replace people, but the truth is, well, not that, which really hampers the sales process. I think we're seeing this with Elon Musk desperately trying to push people into "FSD". People aren't impressed by AIs that aren't as good as people, if not better than people, at doing whatever task they are supposed to do.

_akhe2• 2 years ago

I've made AI assistants that are perfectly accurate with products, pricing, etc. yet still maintain a human quality: https://github.com/bennyschmidt/ragdoll-studio/tree/master/e...

You can accomplish this with RAG.

Your overall point is taken though, the LLM itself is not enough, fine-tuning is not always feasible, and I think no matter how good an AI persona gets at, say, teaching yoga - for some yoga students it will never replace an in-person instructor.

However for a game NPC, online agent, Discord bot, etc. not to mention research, translation, tutorials, summarizing, etc. there is a lot of present day utility for LLMs.

amiantos2• 2 years ago

I find that the level of hype is very correlated to how much the author is involved in making his own LLM chatbot based products, which is not surprising.

Interesting idea with ragdoll, but I'd hate to try to compete with Tavern/Pygmalion cards + Lorebooks, seems like there is a critical mass there already for RP chatbots.

lolinder5• 2 years ago

We're focused on the bad example because it's literally the title of the article and the model's inability to solve that problem has nothing to do with context windows and everything to do with "when all you have is a hammer".

It doesn't matter if the context window is large or small, the Harry Potter Problem as formulated is going to be just as hard because it's not a problem with false advertising in context window sizes, it's a problem inherent to the computing paradigm.

A version of the Harry Potter Problem that was formulated around a model's ability to recall specific scenes of a novel would be much more useful as an illustration of the limitations of the supposedly-large context windows.

araghuvanshi16• 2 years ago

Well the same principle of false advertising re: context window sizes also applies to its inability to count, no? AI companies claim that their models can do math, so wouldn't a regular developer assume that they can also count?

And if I can't trust a so-called SOTA model to partially answer - say, recall each mention of the word "wizard" instead of just giving me the wrong answer - then why should I trust it to list out specific scenes? That's even harder to benchmark.

BeFlatXIII• 2 years ago

It's been sold as a miracle that can replace people, but the truth is, well, not that, which really hampers the sales process

How soon until the hype fades and we enjoy our next AI winter?

Jerrrry3• 2 years ago

Just as notable: my vacuum makes a workable, albeit poor, rake outside.

araghuvanshi16• 2 years ago

Then why do the creators of this vacuum advertise the fact that it's really good at raking? And unlike your analogy, to actually figure out that it's bad at raking you have to read a bunch of academic papers?

doug_durham3• 2 years ago

Where have you read that creator of LLMs say their products are awesome at counting? It's just the opposite.

IgorPartola3• 2 years ago

Isn’t the solution to this particular problem to do what a human would do? I just tried this:

    Write me a short Python program that will count how many times a word occurs in a given text.
    
    Sure, I can help you with that! Here's a short Python program that counts how many times a word occurs in a given text:
    
    ```python
    def count_words(text, word):
        words = > text.lower().split()
        count = 0
        for w in words:
            if w == word.lower():
                count += 1
        return count
    
    text = "Snapchat is a fun way to share moments with friends!"
    word = "Snapchat"

    result = count_words(text, word)
    print(f"The word '{word}' occurs {result} times in the text.")
    ```
    
    You can replace the `text` variable with your own text and the `word` variable with the word you want to count. Let me know if you have any questions!

Seems to me that if you give an LLM an environment to run said program it would be able to automatically do this with the correct prompt. This doesn’t solve the insurance policy problem at all but the solution to these problems is different in my opinion.

araghuvanshi16• 2 years ago

That's true, but the problem of long context understanding (say, "summarize each of the situations where the word 'wizard' is mentioned") remains. And that gets much closer to the insurance policy thing.

IgorPartola3• 2 years ago

“Write me a Python program that extracts context surrounding a word from a long text, then creates a prompt to summarize the context.” Still different than the insurance policy problem.

Scarblac• 2 years ago

At this point, why get the ai involved? The script is trivial and you still need enough knowledge to judge whether it will work, and to run it.

Why doesn't the AI itself decide that writing a Python script is a good way to approach the problem?

seangrogg2• 2 years ago

This feels... a bit obvious to the point of being silly?

It is fairly well-established that context windows are a general issue among LLMs due to SOTA context windows still being somewhere greater than linear. It's also fairly well-established that LLMs aren't necessarily good at things they aren't trained at.

If you are unwilling or unable to throw enough hardware to overcome the context window problem you'll need to reduce the context. If you're unwilling or unable to train the LLM to task you'll have to restructure information such that the task is more tractable.

I'm glad to see that given their constraints they chose a sensible solution for the business, but overall this really seems like a series of known limitations being called out and doesn't feel like it's a good look coming from a company that touts leveraging AI for pulling information from documents and integrating with existing systems...

araghuvanshi16• 2 years ago

I don't think that this is obvious at all. Yes, AI people who read papers on arxiv and know what "SOTA" stands for know it, but that is no longer the main user base of LLMs.

This is meant to be for the developer who doesn't fit the above profile and thinks a model that has a million token context window and "can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks" (direct quote from Anthropic's website), actually can do those things.

seangrogg2• 2 years ago

Valid! I think the disparity is that the article appears to be written for a fairly technical crowd but the expectations appear to come from what these particular models are marketing. Most that are fine-tuning LLMs or aware of LongRoPe for extending context windows are probably consumers of research/white papers rather than marketing material.

Having read some of your other comments it appears that part of the issue is that you were marketed a 1 million token context window and research has shown that's not quite the case. That said, the article doesn't do a good job of painting that picture - it is alluded to with "all fail at this task despite having big context windows" but I think it's worth being crystal clear here that the marketing says 1m and that is disingenuous in your experience and backed by research findings.

lukev• 2 years ago

In the words of Charles Babbage, "I cannot rightly apprehend what confusion of ideas would lead to such a question."

LLMs (by themselves) cannot reliably count. If you expect them to, then you're falling into the common trap of extrapolating a metacognition layer where none exists.

TheCoelacanth• 2 years ago

Mention of that limitation is notably absent in the breathless hype about LLMs.

araghuvanshi16• 2 years ago

Direct quote from Anthropic's website: "Opus -Our most intelligent model, which can handle complex analysis, longer tasks with multiple steps, and higher-order math and coding tasks."

So you tell me: if a regular developer reads the above, how can they surmise that the model which can do higher-order math can't count?

stevenhuang2• 2 years ago

Yes, higher order math does not include arithmetic, that should not be confusing.

LLMs and the Harry Potter problem

Related Stories