killthebuddha

this is not super productive, but not including “Germany has Goethe” in the opening line turned me off immediately

+1. This thread, the thread about documentation, and the thread about turning off Sprites, when taken together, thoroughly illustrate why I'm not currently a Fly user.

I don't have an opinion on the matter, but it's pretty popular. According to [1], Phoenix "was used extensively over the past year" by 2.4% of responders.

[1] https://survey.stackoverflow.co/2025/technology

I really appreciate his iconoclasty right now, but every time I engage with his ideas I come away feeling short changed. I’m always like “there is no such thing as outside the training data”. What’s inside and what’s outside the training data is at least as ill-defined as “what is AGI”.

The answer to (2) is, IMO, "to make computing cheaper". It's interesting to me that this is not the obvious, default answer (it may not be the most actionable answer but IMO it should at least be noted as a way to frame discussions). I think we're at the tail end of computing's artisanal, pre-industrial era where researchers and programmers alike have this latent, tacit view of computing as a kind of arcana.

I feel like the Netflix tech blog has officially jumped the shark.

https://platform.openai.com/docs/guides/agents#tools

I've always felt like the argument is super flimsy because "of course we can _in theory_ do error correction". I've never seen even a semi-rigorous argument that error correction is _theoretically_ impossible. Do you have a link to somewhere where such an argument is made?

Your second paragraph is basically what I'm saying but with the extension that we only actually care about reasoning when we're in these kinds of asymmetric situations. But the asymmetry isn't about the other reasoner, it's about the problem. By definition we only have to reason through something if we can't predict (don't know) the answer.

I think it's important for us to all understand that if we build a machine to do valuable reasoning, we cannot know a priori what it will tell us or what it will do.

Perfect reasoning, with certain assumptions, is perfectly deterministic, but that does not at all imply that it's predictable. In fact we have extremely strong evidence to the contrary (e.g. we have the halting problem).

IMO verifying a solution is a great example of how reasoning is unpredictable. To say "I need to verify this solution" is to say "I do not know whether the solution is correct or not" or "I cannot predict whether the solution is correct or not without reasoning about it first".

One thing he said I think was a profound understatement, and that's that "more reasoning is more unpredictable". I think we should be thinking about reasoning as in some sense exactly the same thing as unpredictability. Or, more specifically, useful reasoning is by definition unpredictable. This framing is important when it comes to, e.g., alignment.

The issue isn’t with who’s hosting, it’s that their SDKs don’t clearly integrate with existing HTTP servers regardless of who’s hosting them. I mean integrate at the source level, of course they could integrate via HTTP call.

I see a good number of comments that seem skeptical or confused about what's going on here or what the value is.

One thing that some people may not realize is that right now there's a MASSIVE amount of effort duplication around developing something that could maybe end up looking like MCP. Everyone building an LLM agent (or pseudo-agent, or whatever) right now is writing a bunch of boilerplate for mapping between message formats, tool specification formats, prompt templating, etc.

Now, having said that, I do feel a little bit like there's a few mistakes being made by Anthropic here. The big one to me is that it seems like they've set the scope too big. For example, why are they shipping standalone clients and servers rather than client/server libraries for all the existing and wildly popular ways to fetch and serve HTTP? When I've seen similar mistakes made (e.g. by LangChain), I assume they're targeting brand new developers who don't realize that they just want to make some HTTP calls.

Another thing that I think adds to the confusion is that, while the boilerplate-ish stuff I mentioned above is annoying, what's REALLY annoying and actually hard is generating a series of contexts using variations of similar prompts in response to errors/anomalies/features detected in generated text. IMO this is how I define "prompt engineering" and it's the actual hard problem we have to solve. By naming the protocol the Model Context Protocol, I assumed they were solving prompt engineering problems (maybe by standardizing common prompting techniques like ReAct, CoT, etc).

Location: San Diego, CA, USA

Remote: preferred but not necessary

Willing to relocate: no

Technologies: TypeScript, React, Next.js, Node.js, Postgres, Docker, AWS, GitHub CI, Python, Elixir, Golang, Java

Résumé/CV: https://www.ktb.pub/dev/resume.pdf

Email: achilles@ktb.pub

I'm a full-stack developer with wide-ranging technical experience and strong general problem solving skills. Most recently I co-founded a startup, worked on it for a few years, and then took some time off to recharge, be with my family, and work on hobby projects. I'm most interested in, and in my opinion best suited for, the kind of fast-paced small-team environment you typically find in early-stage startups.

This looks like a hard maybe, thank you!

Hmm, maybe. I could run a server on my laptop and have the wifi/bt mic stream to it. But I really want this to work on the go, and having a running laptop, or Pi or whatever, at all times is kind of a non-starter.

It's funny how, after years of hearing his voice and not seeing his face, seeing his face puts me in smack in the middle of the uncanny valley.

BLOOD FOR THE BLOOD GOD

Edit: On a more serious note, this book actually changed the way I think in a very concrete (and positive) way. The book spurred me to start consciously reflecting on "what about X, specifically, is surprising" and/or "what about X, specifically, is confusing". It's impossible to quantify, but the habit has definitely, definitely, boosted my ability to disentangle complex situations and make better decisions.

I love Neuromancer specifically for the first third or so, so maybe the latter?

IMO the first part of the book is peak cyberpunk vibes. In particular I read it almost like I would read poetry, late at night when I can't sleep, sometimes jumping back and forth between pages.

if you zoom out far enough, there's no other option (imo)

I think it’s basically “HN is extremely hostile to iconoclastic ideas”

Hmm. My first thought was that I shared non-public links, but I double-checked I can access them from another machine.

I think the post is great, clear and fair and all that. And I definitely agree with the general point that o1 shows some amount of improvement on generality but with a massive tradeoff on cost.

I'm going to think through what I find "misleading" as I write this...

Ok so I guess it's that I wouldn't be surprised at all if we learn that models can improve a ton w.r.t. human-in-the-loop prompt engineering (e.g. ChatGPT) without a commensurate improvement in programmatic prompt engineering.

It's very difficult to get a Python-driven claude-3.5-sonnet agent to solve ARC tasks and it's also very difficult to get claude-3.5-sonnet to solve ARC tasks via the claude.ai UI. The blog post shows that it's also very difficult to get a Python-driven o1-preview agent to solve ARC tasks. From a cursory exploration of o1-preview's capabilities in the ChatGPT UI my intuition is that it's significantly smarter than claude-3.5-sonnet based on how much better it responds to my human-in-the-loop feedback.

So I guess my point is that many people will probably come away from the blog post thinking "there's nothing to see here", o1-preview is more of the same thing, but it seems to me that it's very clearly qualitatively different than previous models.

Aside: This isn't a problem with the blog post at all IMO, we don't need to litter every benchmark post with a million caveats/exceptions/disclaimers/etc.

I agree with basically everything you said but I think you've misunderstood my point. I'll reply to the other comment with more.

Basically yes, but there's a very wide range of how explicit the feedback could be. Here's an example where I tell gpt-4 exactly what the rule is and it still fails:

https://chatgpt.com/share/66e514d3-ca0c-8011-8d1e-43234391a0...

and an example using gpt-4o:

https://chatgpt.com/share/66e515da-a848-8011-987f-71dab56446...

I'd share similar examples using claude-3.5-sonnet but I can't figure out how to do it from the claud.ai ui.

To be clear, my point is not at all that o1 is so incredibly smart. IMO the ARC-AGI puzzles show very clearly how dumb even the most advanced models are. My point is just that o1 does seem to be noticeably better at solving these problems than previous models.

In my opinion this blog post is a little bit misleading about the difference between o1 and earlier models. When I first heard about ARC-AGI (a few months ago, I think) I took a few of the ARC tasks and spent a few hours testing all the most powerful models. I was kind of surprised by how completely the models fell on their faces, even with heavy-handed feedback and various prompting techniques. None of the models came close to solving even the easiest puzzles. So today I tried again with o1-preview, and the model solved (probably the easiest) puzzle without any kind of fancy prompting:

https://chatgpt.com/share/66e4b209-8d98-8011-a0c7-b354a68fab...

Anyways, I'm not trying to make any grand claims about AGI in general, or about ARC-AGI as a benchmark, but I do think that o1 is a leap towards LLM-based solutions to ARC.

A function that implements natural language -> bytecode is IMO way more likely to be under the hood an LLM operating a compiler (or maybe a compiler operating LLMs) rather than a "bare" LLM. From an end user's perspective maybe it won't matter but I think it's an important technical point. IMO there's no evidence that an LLM will ever be the best way to execute general purpose computations.

FWIW this (approximately) is what everybody (approximately) is trying to do.

On the one hand this is interesting to me because it's closely aligned with my own ideas about what will be the general form of the first AGI systems.

On the other hand, just like with my own ideas, there's very little concrete evidence that they're correct. It looks like the author has been very careful about wording and avoids saying directly "there exists an implementation of these ideas that seems to be working well enough to expect it to scale".