robrenaud

Does the model quality become degraded in other ways?

Do consider the incentives of those developing the model that made those predictions. Afaict, it was not selected for purpose other than testimony.

How do you get personalization without surveillance?

I suspect idiomatic TypeScript or idiomatic F# are both way better solutions in the real world than abstruse Typescript emulating idiomatic F#.

If a good SWE is $150/hour, does the model cost actually matter? Surely you'd be willing to spend $10/hour to make that SWE 20% more productive? The model cost is still much less than the salary.

They use a lightweight adapter to silently degrade the performance. Usually these adaptors are made to improve the performance for a given domain/task.

3 blue 1 brown has a great visual introduction to transformers, the heart of LLMs.

It's chapter 5. Start at chapter 1 if you want more background on neural nets and backprop.

https://youtu.be/wjZofJX0v4M?si=HFXbrB-5cArprGaU

"The reasoning is the weights."

The reasoning is in a process that uses the weights.

Sorting algorithms are just bytes. Those bytes don't sort by themselves. They do instruct a computer on how to sort though.

There is some recent work on modularizing knowledge in LLMs.

https://arxiv.org/html/2605.06663v1

It might be possible to train a big generalist that is a composition of modules, some of which can be dropped dynamically at inference time, depending on the prompt.

Is every American tax payer morally compromised?

My big gripe with unions is the unwavering protection of their worst performing members.

Eg, that they necessitated so called "rubber rooms" like these in the NYC public schools, where teachers got paid to do nothing while waiting on arbitration.

https://en.wikipedia.org/wiki/Reassignment_center

The flat earthers are why I hate astronomy.

Afaict, the grand parent poster is just very wrong. You do want to cause acute stresses to your heart (cardiovascular exercise) to get it work better.

Yeah, it's different. Anthropic profits when it delivers tokens. Hosting providers pay when Anthropic scrapes them.

Yeah, my big problem with the paper is it just might be an artifact of qwen's training process.

Was Alphago's move 37 original?

In the last step of training LLMs, reinforcement learning from verified rewards, LLMs are trained to maximize the probability of solving problems using their own output, depending on a reward signal akin to winning in Go. It's not just imitating human written text.

Fwiw, I agree that world models and some kind of learning from interacting with physical reality, rather than massive amounts of digitized gym environments is likely necessary for a breakthrough for AGI.

Recursive self improvement. It's when AI speeds up the development of the next AI.

Location: SF (current). NYC/Philly general area acceptable. Remote okay. email: rrenaud@gmail.com Resume: 16 year SWE -> MLE @ Google, MS from NYU with focus on ML. Retired. Now I hack on data analysis for video game projects for fun, and I love it. I'd take crazy low compensation to do work with interesting game data sets. EG, for game balance, strategic analysis, or to improve/augment game video content.

What do y'all think about the latency/quality tradeoff with LLMs?

Human voices don't take 30 seconds to think, retrieve, research, and summarize a high quality answer. Humans are calibrated in their knowledge, they know what they understand and what they don't. They can converse in real time without bullshitting.

Frontier real time-ish LLM generated voice systems are still plagued by 2024 era LLM nonsense, like the inability to count Rs in strawberry. [1]

I'd personally love a voice interface that, constrained by the technology of today, takes the latency hit to deliver quality.

[1] https://www.instagram.com/reel/DTYBpa7AHSJ/?igsh=MzRlODBiNWF...

Please serve well quantized models.

If you can get 99 percent of the quality for 50 percent of the cost, that is most times a good tradeoff.

Cite a source. Your concrete claim is that, on average, for every $1 of subscription revenue on a monthly subscription, OpenAI and Anthropic were losing $11.50?

It seems completely implausible.

I could believe that if a $20 sub used every possible token granted, it would cost $250. But certainly almost no one was completely milking their subscription. In the same way that no one is streaming netflix literally 24/7.

I used to play very competitively, but I've been more chill recently. I just think it's a nice problem/dataset to work with, because of the depth of my understanding of the game.

I’ve been experimenting with a live win probability predictor for the 10-player arcade game Killer Queen. The goal is to predict the winner in a causal, event-by-event fashion.

Right now I’m struggling to beat a baseline LightGBM model trained on hand-engineered expert features. My attempts at using a win probability head on top of nanoGPT, treating events as tokens, have been significantly worse. I am seeing about 65% accuracy compared to the LightGBM’s 70%. That 5% gap is huge given how stochastic the early game is, and the Transformer is easily 4 OOM more expensive to train.

To bridge the gap, I’m moving to a hybrid approach. I’m feeding those expert features back in as additional tokens or auxiliary loss heads, and I am using the LightGBM model as a teacher for knowledge distillation to provide smoother gradients.

The main priority here is personalized post-game feedback. By tracking sharp swings in win probability, or $\Delta WP$, you can automatically generate high or low-light reels right after a match. It helps players see the exact moment a play was either effective or catastrophic.

There is also a clear application for automated content creation. You can use $\Delta WP$ as a heuristic to identify the actual turning points of a match for YouTube summaries without needing to manually scrub through hours of Twitch footage.

A compiler that can turn cash into improved code without round tripping a human is very cool though. As those steps can get longer and succeed more often in more difficult circumstances, what it means to be a software engineer changes a lot.

Previously, I made a live win probability model for the 5v5 arcade game Killer Queen Arcade from their game events API.

Now I am trying to use that model to make:

1. A post game instant replay that shows the most important/pivotal moments from the most recently finished game. Some arcades have a seperate display for observers, it could work well there, or as good filler between matches on twitch streams.

2. A personalized per tournament/yearly highlights recap.

If it works well, it might be a kind of tool that generalizes well for summarizing long twitch streams for Youtube.

https://github.com/rrenaud/kq_stream_highlights

Here is research about doctors interpreting test results. It seems to favor GP's view that many doctors struggle to weigh test specificity and sensitivity vs disease base rate.

https://bmjopen.bmj.com/content/bmjopen/5/7/e008155.full.pdf

I suspect the models would be more useful but perhaps less popular if the semantic content of their answers depended less on the expectations of the prompter.

What are you embedding? Are you doing a geo restricted area (small universe?).

Paying the local currency with your own cards seems simple and works?

I learned Python circa 2000 as a 17 year old.

It felt pretty easy to read and write, had minimal surprises, and it made writing simple programs easy. The batteries-includedness was great.

It felt like it was designed by a smart guy for practical programming, rather than by a brilliant academic who was wed to purity for maximum elegance. It accepted some warts, but them in mostly ergonomic places.

It was dictated by people who could relegate map, filter, and reduce from builtins to the library. As much as I personally even liked map in particular (I am not super anti functional programming), it's nice to realize the designer had the taste to prefer longer but more explicit programs.

    ys = [f(x) for x in xs]
    ys = map(f, xs)

If you show both of those lines of code to a person in cs101 who has studied Java for a couple months but hasn't seen Python, I am pretty sure they are gonna understand the first line way quicker. Mostly consistent decision making like that leads to ergonomic, practical languages. And they win.

It fits in people's brains better. See also, pytorch vs tensorflow.

Omg, it was so frustrating to say:

Summarize recent working arxiv url

And then it tells me the date is from the future and it simply refuses to fetch the URL.