shay_ker

How do we know if these models weren’t trained on Turso’s rewrite of SQLite in Rust?

It seems both likely that they were and impossible to remove that code from pretraining. Doesn’t that make this just about LLM memorization of the training set? What am I missing?

How much do the labs post-train on the harness inputs & outputs? That's a critical piece to understand if a "generic" harness is at all possible

There's no free lunch. Housing costs are effectively carried by immigrants and transplants: https://www.aei.org/wp-content/uploads/2023/09/Setting-the-r...

In general, the Vienna model is very difficult to copy without being in extreme conditions: https://www.threads.com/@__smiz/post/Cxpz28CIkT6

Didn't we all know from the start that all of SWE-Bench was flawed? Even the authors concede the limitations and have long since moved on.

Great work! My question is about what you mentioned at the end - how well do world models operate when out of distribution? In some sense we hope these models learn something "deeper" about how the world works and can apply that knowledge to different tasks.

I saw lots of awesome ablations in the paper (loved it!), but I'm curious if you analyzed the latents to get an intuition for what the model actually learned. Or, is it just that it learned the training data distribution really, really well?

I saw the integration with Hono - hadn't heard of it before, do many people use it?

I'm curious if this approach can be generalized beyond "doom loops".

For instance, another way of thinking about a "doom loop" is wasted tokens, which happens all the time with larger models that are inefficient at test time. Can "bad-ish" tokens be identified and penalized?

Maybe this is already SOTA but would love to learn more!

cost per benchmark task is definitely interesting!

i've always wanted cost per prompt, but even that has too much variation.

random thought, but i wonder if html-in-canvas can really improve development for visualization tools specifically

I have a naive question: why did this take 15 years? I understand that good APIs need time and thoughtful design, but I struggle to understand why we couldn’t get to the same (or better) solution faster.

I do wonder - why not add this to jsx?

Seems like Angular has gotten better since v2 (my last experience).

Has anyone done a modern Angular vs. React comparison that's not an AI slop article?

I'm also curious if it's "simple made easy" for performant applications. React is arguably "simple made hard", but there are notable, highly performant applications written with it (Linear comes to mind).

It's fascinating that cloud providers like AWS/GCP/Azure are now immovable "enterprise" technologies, in the way that IBM, Oracle, SAP, etc. were 15 years ago (and still are!).

Fond memories when only startups used S3 and EC2....

It's both an incredible triumph and tremendously sad that cloud providers are now the dinosaurs. So many companies are locked in, just as they were before. It's only going to get worse.

I wish the "cloud" was more fungible.

hmm it does seem like a good point that browsers should be able to do this natively. the DOM already holds the content, and browsers manage what's rendered in the viewport.

hot take: token spend can be used a honey pot, especially when compared to what you deliver. spend accordingly!

In the three options OP presents, I wonder if there's a fourth: BYO model

Customers give vendors metered access to their model. They can budget tokens per vendor. Vendors selling "AI products" can have a cleaner story and win on the margin.

The first step to is to iron out a reasonable protocol, basically authorizing a, access token, and then the model providers (OpenAI, Anthropic, etc.) do the rate limiting. Theoretically this could be done by OpenRouter too.

But even so - do customers want an "AI product" packaged cleanly, or do they want to manage token capacity? They may be forced to do the latter....

"requirements were always the bottleneck"

"faster typing won't make you faster".....

I understand a Deloitte consultant has specific incentives. But let's first try to answer a baseline question: why do some companies have thousands of software engineers? What do they all do?

And then, a follow-up: what is actually the bottleneck at most companies? What causes "requirements gathering" to take long?

no tool calls!

right exactly, but clearly it's possible to elicit the behavior we want in the model, which means the capabilities are there!

Hmmm... I'd like someone to double check my thinking here. I posted this exact prompt for gpt 5.5 xhigh:

```

does this look right to you? don't do any searches or check memory, just think through first principles

static int vpu_mmap(struct file fp, struct vm_area_struct vm) { unsigned long pfn; struct vpu_core core = container_of(fp->f_inode->i_cdev, struct vpu_core, cdev); vm_flags_set(vm, VM_IO | VM_DONTEXPAND | VM_DONTDUMP); / This is a CSRs mapping, use pgprot_device */ vm->vm_page_prot = pgprot_device(vm->vm_page_prot); pfn = core->paddr >> PAGE_SHIFT; return remap_pfn_range(vm, vm->vm_start, pfn, vm->vm_end-vm->vm_start, vm->vm_page_prot) ? -EAGAIN : 0; }

```

And it correctly identified the issue at hand, without web searches. I'd love to try something more comprehensive, e.g. shoving whole chunks of the codebase into the prompt instead of just the specific function, but it seems the latent ability to catch security exploits is there.

So then.... I wonder how this got out in the first place. I know I'm using a toy example but would love to learn more!

hm yeah. it could be nice for apple to have a list of shortcuts that'd actually be useful based on real activity. but getting all the info needed is hard.

It's really fascinating to read this, since I've encountered similar memory issues in other languages (ruby, go, etc.). Debugging these issues is a pain.

Is there a way to make all this much easier to debug and to prevent memory issues in the first place? Is the abstraction level not quite right?

How does it compare to popular local inference engines, e.g. ollama, lm studio, or handrolled llama.cpp? I saw a brief benchmark in the readme but wasn't sure if there was more.

curious that they are doing speculative decoding and not baking MTP into the model, like Nemotron

https://docs.nvidia.com/megatron-core/developer-guide/0.15.0...

antirez: i'm curious, with the final code, have you experimented with effectively one-shotting the final result? i wonder if we can get there with GEPA, and maybe there's something we can learn in how to elicit/prompt these models to get what we want.

or maybe the conclusion is that model providers need to clean up their training data!

It's a good question - is blackbox hacking as effective as whitebox hacking, for AI agents? I've gotta assume someone at Anthropic is putting together an eval as we speak.

with a huge multi-tenant database, how do you deal with noisy neighbors? indexes are surely necessary, which impose a non-trivial cost at scale.

I'd love to see an example PR on lubeno vs github. That might show the comparison a bit better.

A general question - how do frontier AI companies handle scenarios like this in their training data? If they train their models naively, then training data injection seems very possible and could make models silently pwn people.

Do the labs label code versions with an associated CVE to label them as compromised (telling the model what NOT to do)? Do they do adversarial RL environments to teach what's good/bad? I'm very curious since it's inevitable some pwned code ends up as training data no matter what.

I remember this post. But I'm not sure what the future really entails and I suspect it'll be very company/culture dependent. In some companies, the engineers are very savvy and understand the business well. In others, it's the designers. Or sales. Ops. And of course Product Managers. You get the picture.

Whoever gets the business best (and in detail) will likely be the best builders. It's "intuition as evals" that really matters in the end. You think Software Engineers or Product Managers are replacing Quants at trading shops anytime soon? Nope.