virgilp

The honest title is "what Claude learned from reimplementing 40 Multi-Agent LLM Papers"

Well before I knew he was a nutter, it strikes be that the graph starts in 1991. Feels like we should should have more data than that, and that the same graph would be more powerful if it started in 1800 and showed the same thing (or at least, earlier in the 20th century, surely we must have data from the 1950s at least? probably earlier?)

Not saying that the data is not concerning, or that climate warming isn't real, BTW. I'm just saying this particular graph feels fishy.

It seems that pass rate decreases with effort increase, on GPT5.5? This is highly counter-intuitive and I don't see any explanation, any idea why they'd get this result?

Perhaps you're a better-than-average driver? I unfortunately am not, and have had the opposite experience (people flashing me because I forgot to turn off the high beams, or couldn't do it fast enough due to e.g. a corner, or drinking water with that hand, or whatever). In long straight lines it's relatively easy to do it, on windy roads not so much. On the other hand, with current car I don't remember anybody flashing me.

Personally, I just developed a strategy for dealing with too-bright-incoming-lights: just focus on the right edge of the road - that takes my eyes off the lights, and it's where the danger lies anyway. With that, late-commuting vehicles are a lot less of a nuisance than people who just have poorly tuned low beams or simply keep high beams on always. And those problems/ people are unrelated to the "automatic commute" technology - arguably the tech improves things on that front, even in the not-so-great implementations.

Admittedly mine is somewhat high end, and I have seen broken implementations (which is what I think you describe; e.g. for auto high beam assist mine will redirect the beams around isolated cars, won't completely switch to low beams except in heavy /heavier traffic; and when it does, it takes at least several seconds to switch back - which makes it impossible to do the kind of "flash" you describe).

But I'm kinda' surprised by the cruise control, I don't think I ever drove a car, even a rental, where cruise control wasn't at least "genuinely useful". Even the non-automatic one. How does it "not take into account vehicles in other lanes"/ what makes it dangerous?

If you don't call an exterminator with the proper poison almost any effort you make will be moot.

Nah, not true. I lived in a student housing that was positively _infested_ with cockroaches (and had stuff like wood paneling on the walls, just to get an idea - i.e. lots of places for roaches to hide). We managed to largely get rid of cockroaches in our room (you still get the occasional one, because well, you had to open the door, and hallways were infested as I mentioned).

It's not _that_ hard, there are a lot of solutions. You need to do 3 things:

a. seal all holes/cracks/niches (e.g. with silicone). cover ventilation holes with nets. Install sponge/rubber bands to make sure doors/windows close well.

b. kill them once when you move in (after doing the work at point a) using copious amounts of insecticides; then install roach traps (sticky ones are good) to catch the occasional one that makes it through your defenses. Keep occasional spraying in the corners/ behind the fridge/ near the pipes/ in places where they are likely to gather.

c. Keep it clean/ don't offer a lot of incentives for roaches to come over to you (no breadcrumbs all over the place, food in closed containers etc)

Do these well and you should be largely roach-free, regardless of the building. But yeah, it's an annoying fight if the building itself is infested.

So do other drivers that fail to switch to low beams - even my old car was better than me in switching to low beams.

How are all these "broken and dangerous"? In my car (Volvo) they work rather well. Perhaps sign detection sometimes misses signs, but so do I so I can't fault it. The others though, I rank them all somewhere between "genuinely useful" and "absolutely awesome"

He talks about 15% _per month_

You can still believe that the scientific method works; and might leads you to 2 conclusions:

(a) "I can prove earth is not flat" (using this methodology) (b) I cannot prove there is no God, though I may believe the prevalence of evidence does not support the hypothesis, there's no scientific test that I can design.

To be fair, their control variables treat the first objection (wealth), not the second (brand preference; and yeah there's some correlation but one doesn't imply the other)

That's the 4th option

That's for the good studies. Let's not pretend that all published studies are honest. Unfortunately it is quite reasonable to be skeptical about extraordinary claims such as this one.

If we ignore cost (which is kinda hard to ignore), I feel Codex kinda' does it for me. Sure it's not really an editor but I find I don't need that _that much_ and it's easy to launch an external editor (they actually have the feature).

The ironic thing is that half a year ago, after trying factory.ai I thought chat-first interface was a stupid idea that will never work.

I wonder how knowledgeable in compilation was the engineer that attempted this. I'm pretty confident that I could produce a decent C compiler in a few weeks (or less), if given Opus 4.7 + unlimited tokens + a good test suite. (and this is not blind unsubstantiated belief in AI, I've recently rewritten a somewhat sophisticated interpreter in a week with AI; and have worked on several C++ compilers in the past, including a GCC port to a custom DSP, so I have a bit of an idea about what this would take).

But yeah, this is not a "one shot" project, none of it is. One shot doesn't work even with humans - after all, this is exactly what killed waterfall as a methodology.

you can absolutely know. they do suspiciously well. you just give harder problems until they can't solve it. how they react/approach a problem that they can't immediately solve _is_ the interview - not the "how many things they solved correctly" part.

That said - I seldom need people to be hardcore algorithm solvers What I typically did was a variation of fizzbuzz (can the candidate code very basic logic?) and then finding a bug or minor requirements extension in their online screening test/"homework" and asking them to solve that on the spot (did they write the code themselves/can they modify it). It's typically enough, there's diminishing returns to test more in-depth the programming skills - the rest you can discuss domain knowledge, general experience, working style etc.

To be fair it doesn't say "you can't score a goal" or "you can't kick the ball", it says you can only decide to _try_ to do that. But agree it's not that deep as they seem to think, you can take this line of thinking further as much as you want (you can try to move your leg but perhaps due to injury or external blockage it won't move; so all you can really do is "desire" not "decide"? Is there some very deep meaning to this? And what even is this "you" that we are talking about?)

One thing that is worth pondering is what parts of the "old wisdom" (if any) are no longer true. Because the set of "common sense knowledge" has a tendency to mutate in time. Take the first statement:

Impactful software tends to be written by many humans that need to collaborate.

This was definitely true. Is it still true to the same extent/ in the same way? Not obvious...

In Romania I think they just gave back the money (or maybe it was on a voucher with "if you don't use the voucher by date X, we'll refund the money"). which is in stark contrast with how other low-cost airlines like WizzAir behaved. Perhaps it was regional policy; or perhaps it was due to their previous interactions with UK regulators? But for me, they gained a lot of respect for them back then (whereas WizzAir is on the "only if absolutely no other choice" list - and I think I only used it once, for a business trip where it had a good direct flight AND I didn't care if I actually made it to the destination, or if I got stranded there for a few days - since the company would've been paying)

I kinda' like Ryanair as lowcost airline? They're fairly efficient (boarding, serving etc), they _actually fly_ the advertised flights (with relatively few exceptions), and the food is reasonably priced. During COVID they would just give your money back, no shenanigans like "they're in our company wallet". Sure they have their quirks but they don't seem to go out of their way to deceive you, they're pretty open about what you pay and what you get.

Now Wizzair is "mostly not an airline" for me, because they have all the negative traits I hinted above. E.g. they'll happily advertise flights they have no intention of flying, make refunds hard, are as misleading as they can be about pricing, make it impossible to checkin online a few hours before the flight so that you have to pay their high fees, etc.

I wouldn't want the Ryanair experience for long-haul flights; but for short 2-3h ones within Europe, they're fine, I'm always considering them. Not for the perceived cheapness, but for the "I expect them to actually fly AND be on time" part.

If your feedback loop is hours or days, I don't think it's bad you spend some time thinking ahead of doing. Oh, you missed the unknown unknowns? You'll hit them soon enough anyway, this is not a model that encourages abstract planning with no action taken, for extended periods of time....

The problem with "AI zealots" is seldom that they spend too much time planning ahead. If anything, it's the opposite.

Waterfall was bad due to the excessively long feedback loops (months-to-years from "planning" to "customer gets to see it/ we receive feedback on it"). It was NOT bad because it forced people to think before writing code! That part we should recover, it's not problematic at all.

qa has long ago merged with programming in "unified engineering". Also with SRE ("devops") and now the trend is to merge with CSE and product management too ("product mindset", forward-deployed engineers). So yeah, pretty much, that's the trend. What would you trust more - an engineer doing project management too - or a project manager doing the engineering job?

I honestly don't see how this is related? Nothing says "one shot a full system from a perfect specification", I don't think this was ever a goal (or that it will be practical to do so)

Actually, no. We always needed good checks - that's why you have techniques like automated canary analysis, extensive testing, checking for coverage - these are forms of "executable oracles". If you wanted to be able to do continuous deployment - you had to be very thorough in your validation.

LLMs just take this to the extreme. You can no longer rely on human code reviews (well you can but you give away all the LLM advantages) so then if you take out "human judgement" *from validation*[1], you have to resort to very sophisticated automated validation. This is it - it's not about "inventing a new language", it's about being much more thorough (and innovative, and efficient) in the validation process.

[1] never from design, or specification - you shouldn't outsource that to AI, I don't think we're close to an AI that can do that even moderately effective without human help.

Also: if that one particular AI-produced compiler has nothing innovative, that only means that the human "director" behind the AI didn't ask it to produce anything innovative; what it does not mean is that AI can never produce anything innovative in a compiler.

"Waterfall" got a bad rep because it meant "we stay months in the requirements gathering, then months design phase, then months in development, then months in validation". If you compress "months" to days/hours, what you obtain is something that nobody from the 90s would recognize as "waterfall"; it is not the end of agility, far from it.

Cool but it is not a framework for working with AI, it is an _opinionated_ framework for building full-stack apps right? As in, I can't use any of it if I'm building, say, a Spark data processing pipeline. Or a ML framework. Or automation software that runs on custom processors.

The idea of "guardrails outside the model" is definitely appealing but I wonder if you can make it generalize well.

No, OP literally claims "you can't spec out something you have no clue how to build"; I claim that on the contrary, you absolutely can - you don't need to know "how to build" but you need to clarify what you want to build. You can't ask AI to build something (and actually obtain a good "something") until you can say exactly what the said "something" is.

You iterate, yes - sometimes because the AI gets it wrong; and sometimes because you got it wrong (or didn't say exactly what you wanted, and AI assumed you wanted something else). But the less specific and clear you are in your requirements, the less likely it is you'll actually get what you want. With you not being specific in the requirements, it only really works if you want something that lots of people are building/have built before, because that will allow the AI to make correct assumptions about what to build.

Nothing of what you write here matches my experience with AI.

Specification is worth writing (and spending a lot more time on than implementation) because it's the part that you can still control, fully read, understand etc. Once it gets into the code, reviewing it will be a lot harder, and if you insist on reviewing everything it'll slow things down to your speed.

If the cost of writing code is approaching zero, there's no point investing resources to perfect a system in one shot.

THe AI won't get the perfect system in one shot, far from it! And especially not from sloppy initial requirements that leave a lot of edge (or not-so-edge) cases unadressed. But if you have a good requirement to start with, you have a chance to correct the AI, keep it on track; you have something to go back to and ask other AI, "is this implementation conforming to the spec or did it miss things?"

five different versions of the thing you're building and simply pick the best one.

Problem is, what if the best one is still not good enough? Then what? You do 50? They might all be bad. You need a way to iterate to convergence