The One-Step Trap (In AI Research)

http://incompleteideas.net/IncIdeas/OneStepTrap.html

by jxmorris12 • 10 days ago

53 10 10 days ago

ssivark2• 10 days ago

Ha, interesting. I wasn't aware of Sutton's blog post, but if I might make a shameless plug, we demonstrated [1] exactly this problem (see section 4.4.3), and how multi-step world models (using diffusion models as the substrate) could be one potential answer.

Since then, I have come to like temporally-abstract models more and more. Rolling out in time -- either step-by-step or many steps at once -- suffers from the tyranny of the specific. For long horizon planning with agents, I care (often only approximately) about where I can end up, and seldom about exactly when I end up there. Successor features, GVFs, Forward-Backward representations, and the like seem like they have an elegant approach for structuring thinking at a "high level", instead of generating exponentially large search trees by rolling out microscopic world models.

[1] https://arxiv.org/abs/2410.05364 (funnily, from around the same time / few months after Sutton's blog post)

sawyers• 10 days ago

What do you mean by tyranny of the specific?

ssivark2• 10 days ago

Imagine I want to attend a conference in a different country. Google maps might give turn by turn navigation but that is an overwhelming and largely irrelevant mess of details for most planning purposes. Eg: all I might want to know is the different flight legs and the fact that the journey takes 15-18 hours, and not all the turns and traffic lights to get from home to the airport.

I want a zoomed out picture, and to be able to fill in detail hierarchically, on demand. Instead, one-step models give you the full high-res local structure of the graph that would have to search through (with too many states and edges).

mxwsn• 10 days ago

This is the same reasoning behind why Yann Lecun thought test-time scaling would not work for LLMs: compounding error.

Instead, the more tokens LLMs use, the better their performance on many tasks. LLMs can self-correct, evidenced by the power of getting models to question themselves by emitting "Wait," in S1. https://arxiv.org/abs/2501.19393

rf15• 10 days ago

You wouldn't believe the amount of reasoning I saw these past few months that was correct until the stochastic parrot decided that a "wait" token should now be used and everything steered off a cliff.

tipsytoad• 10 days ago

Yeah came here to comment exactly this. And this is generally why I dislike/avoid this type of first principle analysis: it can make very convincing arguments that are just totally wrong due to some misleading assumption

gnabgib• 10 days ago

(2024)

fny• 10 days ago

I'm not sure I follow what one step means exactly. Aren't all models some f(x) = y? Is the suggestion instead that we should be doing f(x) = g(h(x)) = y?

What would the difference be?

smokedetector1• 10 days ago

The fallacy is that f(t+N) can be obtained by iterating f(t+1) N times. This is the “step”

thornewolf• 10 days ago

just read this when reviewing OpenAI's "spinning up" documentation as it was linked there!

The One-Step Trap (In AI Research)

Related Stories