kimjune01

understatement is a kind of humor, when he said 'redditor'

it would be nice if these benchmark reports actually specified which tasks they passed and which ones they didn't.

Chat data is the closest we ever got to purchase intent than demographics or keywords, attribution given a market position is already solved

the investors want their money to make more money

I do this

Attribution would be as easy as knowing which chatbot they came from? Unless chatbot publishers (openai in this case) allows you to target based on the attribution, that extra data will not be actionable, anyway.

the long tail of chatbot users will never pay for subscriptions, that's been proven since WhatsApp tried to charge a dollar or two per year per user

In a chatbot-first world, it's gonna have to either be generic impression ads that race to the bottom or contextual recommendations. Of course, the fear is that the advertiser hoards personal data or the chat answers are skewed to make you spend more money.

Those concerns can be resolved by decoupling the text generation from the recommendation engine. It works by casting your intent and match it up with an advertiser's in a secured environment where the advertiser doesn't actually get to see your chats. For example, if you are talking about knee pain during trail running, the recommendation engine might cast "knee pain for runners" and either a shoe company or a physiotherapist would bid against that in an embedding space.

Because the recommendation engine is an opt-in module for publishers and users, it's game-theory welfare maximizing.

I talk a lot more about this on my blog, it goes on for quite a while. https://www.june.kim/vector-space

The exchange is live, with no publishers or advertisers. https://vectorspace.exchange/

and LLMs are really good at filling in the blanks. It's like a really big import on generic application knowledge, and if you draw the circles well enough, it will draw the rest of the owl.

eval shops who depend on these things for a living will have to version their datasets, to prevent contamination

i meant to make the distinction between 'prompt' and 'spec' in that prompts are something that a human types in by hand. Interesting programs are a combination of canon and novel input

whether the hypothesis originates at the users or in your guess for the users is irrelevant in that they're both hypotheses with no certain outcome until trial.

I agree with iteration, but that's an orthogonal argument to how effective it is at turning prose to code. You are asking, LLMs are capable of translating a 835-page spec, but where does the spec come from? Follow the stream up from the built software product:

Binary < IR < Code < Spec < Prompt (aka goals)

For LLMs to faithfully fulfill the role of translating spec/prompt to code, it needs enough information about the problem/motivation and the constraints around it. Often, that's done by research or by importing proven modules. The human role left in this compilation chain is a precise-enough expression of the motivation, upstream of the built artifacts.

Edit For example, SWEbench Pro was supposed to be a measure for how good LLMs are at compiling prose into code. At least 15% of those tasks were underdetermined by ambiguous prose in the spec. Of the issues that are specific enough, Claude and Codex indeed do compile to functional code with self-written tests. Source: https://github.com/kimjune01/swebench-pro-audit

prompts are not specs because most often, prompts are underspecified. You can prompt fizzbuzz well enough for it to be functional, but not ffmpeg. So the distinction is less about Claude's capability, more about the quality of the input. Garbage in garbage out

a 17 year old can take a nude selfie and be charged as an adult in possession of CSAM

i thought this was what openclaw was for?

i found the same thing with Rust, verification steps are really helpful for speed and correctnesss

lean compiles or it doesnt

this is known as the oracle problem

one of those words that lost meaning, like samaritan or literally

In the days when Sussman was a novice, Minsky once came to him as he sat hacking at the PDP-6.

"What are you doing?", asked Minsky.

"I am training a randomly wired neural net to play Tic-tac-toe", Sussman replied.

"Why is the net wired randomly?", asked Minsky.

"I do not want it to have any preconceptions of how to play", Sussman said.

Minsky then shut his eyes.

"Why do you close your eyes?" Sussman asked his teacher.

"So that the room will be empty."

At that moment, Sussman was enlightened.

oh this again, i found this on stumbleupon circa 2007

using AI to red-team your thoughts and assumptions is the fastest way to get smart since the dawn of time

coopetition

numpy is the APL reincarnate

I appealed one time, took 3 weeks to get my 30 dollars back. will never deal with that again

I've been running a public experiment on how far I can take agentic coding harnesses. When I ran out of my own stuff to work on, I looked to open source.

So far, I got the harness to create 368 PRs, 110 of which were resolved. Of those, I got a 56% merge rate.

The harness: https://github.com/kimjune01/sweep

But as soon as I realized what I was doing to maintainers, I worked all night to build the same pipe but backwards for PR review.

The defense: https://github.com/kimjune01/immune

Which means: bots on the contributor side, bots on the maintainer side. The surprise? What's left is two humans on either end. I accidentally invented a H2H protocol.

I'm happy to share what I learned.

Although Arena is adversarial and resistant to goodharting, it's not immune. Models that train on Arena converge on helpfulness, not necessarily truthiness

are merged PRs a measure of velocity? github.com/kimjune01/

we lack the shared data structure for it