derekcheng08

They have a public dataset of 6000, real world coding agent sessions. About 80% of them are Claude Code. Lots of cool analyses to be done on these.

I built Prism this morning. It runs a tiny 500M parameter LLM in your browser on a given piece of text and visualizes the entropy of the probability distribution computed for each token -- effectively how confident the model is in predicting each token.

I've been wanting to build this for a while. I had a crude version of this when I first started working with LLMs and it really helped with my intuition of how the model worked. When you run it on a block of code, you'll see that the model is unsure when it needs to pick an identifier or start a new line. It's a fascinating glimpse into how models operate -- what is "easy" for them and what is uncertain.

After analyzing thousands of agent trajectories, we’ve seen all sorts of bizarre LLM quirks, this being one of them. Would love to hear other war stories also!

Pretty cool! Always struck me as odd that the IDE has largely remained single-player for 40 years.

That was an insane time. The pace was unreal. I remember Netscape 2.0 had at least 6-7 beta releases prior to the full release. And each one just dropped something massive and fundamental to the Internet - JavaScript (then called LiveScript IIRC) being one of those things. Just casually dropping what would dominate the entire industry in a browser beta.

The only other period I have experienced that comes close is what is happening now. What an incredible time to build.

Super interesting how this arc has played out for Microsoft. They went from having this massive advantage in being an early OpenAI partner with early access to their models to largely losing the consumer AI space: Copilot is almost never mentioned in the same breath as Claude and ChatGPT. Though I guess their huge stake in OpenAI will still pay out massively from a valuation perspective.

The biggest omission that immediately stands out to me is: "provides a clear sense of direction".

I've seen so many examples of teams and organizations that experience a lack of clarity, with all sorts of negative downstream consequences - muddled strategies, moving goalposts, fatigue/low morale. Having a leader that can provide that clarity is so important.

I do really like Tonkotsu :)

But beyond that, we wanted a playful, accessible brand. We think dev tools (particularly ones like Tonkotsu) are consumer products and we didn't want the staid/corporate branding many tools have.

Forgot to mention that an interesting behavior we see emerging is “human as editor”: let the agents make several commits in a branch, and then the human does a single refinement pass over it before raising a PR. Curious if others use this workflow or something else?

Is it model quality or the CLI itself?

Cool, but at the same time, it feels overwhelming: so many different CLI or IDE tools, so many extension points. It will be fascinating to see how this all shakes out.

Really feels like computer use models may be vertical agent killers once they get good enough. Many knowledge work domains boil down to: use a web app, send an email. (e.g. recruiting, sales outreach)

Fascinatingly deep study. It shows the hyperoptimization needed to build these businesses: from all the work needed to calibrate pricing for each country, to technical safeguards like the fingerprinting. A lot of work had to be done here.

I suspect there are many, many things for which chat is a great interface. And by positioning ChatGPT as the distributor for all these things, they get to be the new Google. But you're also right that many domains for which a purpose-built interface is the right approach, and if the domain is valuable enough, it'll have someone coming after it to build that.

What a fun blast from the past! I fondly remember real-time at Waterloo. If nothing else, it taught you how to project-manage/time-manage so you could write a project of tens of thousands of lines of code in a four month term.

Just based on usage, I would assume js/ts would be very valuable as well. I see a lot of the same issues there and agree the core algos seem to apply generally. Very cool project!

LOL came here to say this exactly. Everyone LOVES war stories in my experience :)

This is pretty awesome! If it's built on tree-sitter, is it fair to assume it's generalizable across languages?

AI is (or has the potential to be) a gigantic abstraction layer and software engineering is filled with abstraction layers. But one thing that has consistently held true is that the best engineers -- while taking advantage of abstractions -- also have the curiosity and intelligence to peel back and at least understand the gist of what's going on behind the scenes. So while most of us will never write a TCP/IP stack, it's helpful to know the protocol. While many will simply call into a hosted distributed database, strong engineers will know broadly how it is implemented and there are availability/consistency trade-offs, etc.

It's the same here: if you just shut off your brain and do what AI says, copy/pasting stuff from/to chat windows, that's going to be a bad time.

Interesting. I wonder if you could implement tool calling with this approach so the LLM calls the tool with the formal specification and gets back the result. Just like a coding agent can run a compiler, get back errors and then self-correct.

I suspect in this case, enterprise adoption of chat-based AI is downstream from consumer adoption. Like many things, trendsetting/taste-making happens on the consumer side and then comes to enterprise. ChatGPT was first to the consumer consciousness, while Claude made its mark as most tasteful... Copilot just didn't register.