cornel_io

That's what focus looks like to a just-say-yes engineer. They're just focused on shipping, not tinkering.

Yes, it can. So can I. But neither of us will write the code exactly the way nitpicky PR reviewer #2 demands it be written unless he makes his preferences clear somewhere. Even at a nitpick-hellhole like Google that's mostly codified into a massive number of readability rules, which can be found and followed in theory. Elsewhere, most reviewer preferences are just individual quirks that you have to pick up on over time, and that's the kind of stuff that neither new employees nor Claude will ever possibly be able to get right in a one-shot manner.

Just tell Claude to create tmux sessions for each, it can figure out the rest.

As an investor in Anthropic, I'd say that anyone who wasn't aware of where they stood on various values issues the whole time should not have been putting money in, it was not hidden.

Asking questions on SO was an exercise in frustration, not "interacting with peers". I've never once had a productive interaction there, everything I've ever asked was either closed for dumb reasons or not answered at all. The library of past answers was more useful, but fell off hard for more recent tech, I assume because people all were having the same frustrations as I was and just stopped going there to ask anything.

I have plenty of real peers I interact with, I do not need that noise when I just need a quick answer to a technical question. LLMs are fantastic for this use case.

This article claims that's false, that 8-12 at higher weight leads to the same result as 20+ at lower weights.

And at the end of the day it's not really a tradeoff we'll need to make, anyways: my experience with e.g. Claude Code is that every model iteration gets much better at avoiding balls of mud, even without tons of manual guidance and pleading.

I get that even now it's very easy to let stuff get out of hand if you aren't paying close attention yourself to the actual code, so people assume that it's some fundamental limitation of all LLMs. But it's not, much like 6 fingered hands was just a temporary state, not anything deep or necessary that was enforced by the diffusion architecture.

Theoretical "proofs" of limitations like this are always unhelpful because they're too broad, and apply just as well to humans as they do to LLMs. The result is true but it doesn't actually apply any limitation that matters.

When a tool call completes the result is sent back to the LLM to decide what to do next, that's where it can decide to go do other stuff before returning a final answer. Sometimes people use structured outputs or tool calls to explicitly have the LLM decide when it's done, or allow it to send intermediate messages for logging to the user. But the simple loop there lets the LLM do plenty of it has good tools.

There are thousands of projects out there that use mocks for various reasons, some good, some bad, some ugly. But it doesn't matter: most engineers on those projects do not have the option to go another direction, they have to push forward.

Of course. But I've screened far more out because I was in a rush and got 40 resumes in that day and they just didn't pique my interest as much as the next one over.

There may be a ceiling, sure. It's overwhelmingly unlikely that it's just about where humans ended up, though.

I'm all for benchmarks that push the field forward, but ARC problems seem to be difficult for reasons having less to do with intelligence and more about having a text system that works reliably with rasterized pixel data presented line by line. Most people would score 0 on it if they were shown the data the way an LLM sees it, these problems only seem easy to us because there are visualizers slapped on top.

If ChatGPT claims arsenic to be a tasty snack, OpenAI adds a p0 eval and snuffs that behavior out of all future generations of ChatGPT. Viewed vaguely in faux genetic terms, the "tasty arsenic gene" has been quickly wiped out of the population, never to return.

Evolution is much less brutal and efficient. To you death matters a lot more than being trained to avoid a response does to ChatGPT, but from the point of view of the "tasty arsenic" behavior, it's the same.

Even though this post says exactly the thing that most Proper Analysts will say, and write long LinkedIn posts about where other Proper Analysts congratulate them on standing up for Proper Analysis in the face of Evil And Stupid Business Dummies who just want to make bad decisions based on too little data, it's wrong. The Jedi Bell Curve meme is in full effect on this topic, and I say this as someone who took years to get over the midwit hump and correct my mistaken beliefs.

The business reality is, you aren't Google. You can't collect a hundred million data points for each experiment that you run so that you can reliably pick out 0.1% effects. Most experiments will have a much shorter window than any analyst wants them to, and will have far too few users, with no option to let them run longer. You still have to make a damned decision, now, and move on to the next feature (which will also be tested in a heavily underpowered manner).

Posts like this say that you should be really, REALLY careful about this, and apply Bonferonni corrections and make sure you're not "peeking" (or if you do peek, apply corrections that are even more conservative), preregister, etc. All the math is fine, sure. But if you take this very seriously and are in the situation that most startups are in where the data is extremely thin and you need to move extremely fast, the end result is that you should reject almost every experiment (and if you're leaning on tests, every feature). That's the "correct" decision, academically, because most features lie in the sub 5% impact range on almost any metric you care about, and with a small number of users you'll never have enough power to pick out effects that small (typically you'd want maybe 100k, depending on the metric you're looking at, and YOU probably have a fraction of that many users).

But obviously the right move is not to just never change the product because you can't prove that the changes are good - that's effectively applying a very strong prior in favor of the control group, and that's problematic. Nor should you just roll out whatever crap your product people throw at the wall: while there is a slight bias in most experiments in favor of the variant, it's very slight, so your feature designers are probably building harmful stuff about half the time. You should apply some filter to make sure they're helping the product and not just doing a random walk through design space.

The best simple strategy in a real world where most effect sizes are small and you never have the option to gather more data really is to do the dumb thing: run experiments for as long as you can, pick whichever variant seems like it's winning, rinse and repeat.

Yes, you're going to be picking the wrong variant way more often than your analysts would prefer, but that's way better than never changing the product or holding out for the very few hugely impactful changes that you are properly powered for. On average, over the long run, blindly picking the bigger number will stack small changes, and while a lot of those will turn out to be negative, your testing will bias somewhat in favor of positive ones and add up over time. And this strategy will provably beat one that does Proper Statistics and demands 95% confidence or whatever equivalent Bayesian criteria you use, because it leaves room to accept the small improvements that make up the vast majority of feature space.

There's an equivalent and perhaps simpler way to justify this, which is to throw out the group labels: if we didn't know which one was the control and we had to pick which option was better, then quite obviously, regardless of how much data we have, we just pick the one that shows better results in the sample we have. Including if there's just a single user in each group! In an early product, this is TOTALLY REASONABLE, because your current product sucks, and you have no reason to think that the way it is should not be messed with. Late lifecycle products probably have some Chesterton's fence stuff going on, so maybe there's more of an argument to privilege the control, but those types of products should have enough users to run properly powered tests.

I think you're being downvoted because you're agreeing with the post you're quoting, but arguing as if they're wrong: the example in question was there to show how DI can be useful, so there's nothing to argue against.

I've often run multiple Claude Code sessions in parallel to do different tasks. Burns money like crazy, but if you can handle wrangling them all there's much less sitting and waiting for output.

None of that seems at all user-hostile to me, it's literally all aimed at making sure what the user is shown is more likely to actually be useful to them.

I guess this is a big and probably unbridgeable divide, some people think this sort of thing is obviously evil and others, like me, actually prefer it very strongly over a world where all advertising is untargeted but there is massively more of it because it's so much less valuable...

There are a couple important things to also keep in mind:

First: just like there can be individuals who lift up an entire team but are not ticking off tasks themselves, there can be apparently individually productive team members who slow the entire team down for any of a number of reasons. In my experience it's usually either that they are a) fast but terrible, b) have really weird coding styles that might not be bad but are very difficult for others to work with (architecture astronauts and people who barely speak the language often fall here), or c) are process bullies who try to set up the entire review system to enforce their preferences in a hardline way that suits them but delays everyone but them. Each needs to be dealt with very differently, to varying degrees of success, but my honest opinion at this stage is that no matter how productive these people seem by themselves it's mostly harmful to have them on a team. Behavioral issues in senior people tend to be really tough to adjust, and take a lot of energy from a manager that is better spent helping your top performers excel; that said, if you can get them to adjust sometimes it's worth the effort.

Second: pair programming works great for some people, but it is terrible for others. I've measured this on teams by trial and error over fairly long periods, and unfortunately it's also the case that people don't segment neatly based on their preferences, so the obvious "let them choose" method isn't ideal. There are pairs of people who really love pair programming and desperately want to do it all the time who are almost fully 2x as productive when split apart instead (yes, including downstream bugs and follow-ons, meaning that they really are just having two people do the job of one), and there are people who hate pairing who see similar multiples by being forced into it even though they hate it. My rough intuition is that there are two independent variables here, a social factor that measures how much you enjoy pairing, and a style factor that determines how much benefit you derive from it, with the two not correlating much at all. There might even be a slight anticorrelation, because the more social people who love it have already naturally done as much knowledge sharing as is helpful, and the people who hate it are underinvested there and could use some more focus on the fact that they're part of a team.

There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong:

https://arxiv.org/abs/2402.09733

https://arxiv.org/abs/2305.18248

https://www.ox.ac.uk/news/2024-06-20-major-research-hallucin...

So I don't think it's that they have no concept of correctness, they do, but it's not strong enough. We're probably just not training them in ways that optimize for that over other desirable qualities, at least aggressively enough.

It's also clear to anyone who has used many different models over the years that the amount of hallucination goes down as the models get better, even without any special attention being (apparently) paid to that problem. GPT 3.5 was REALLY bad about this stuff, but 4o and o1 are at least mediocre. So it may be that it's just one of the tougher things for a model to figure out, even if it's possible with massive capacity and compute. But I'd say it's very clear that we're not in the world Gary Marcus wishes we were in, where there's some hard and fundamental limitation that keeps a transformer network from having the capability to be more truthful as a it gets better; rather, like all aspects, we just aren't as far along as we'd prefer.

It's a bit shocking to me: I still remember all the concepts quite clearly from when I studied Galois theory ~20 years ago, to the point where I can run through a lot of the proofs conceptually in my head, but the vocabulary is GONE. Like, completely a blank, I don't remember almost any of the abstract algebra terms that all of this is expressed in.

It reminds me of the truth of the advice that my category theory professor gave us, that the definitions are both the least important and the most important things, simultaneously. They're the least important in that they're just words that wrap up very simple concepts, and merely knowing the definitions doesn't actually mean you can work with the concepts. But they're the most important thing in that most of higher level math really boils down to picking out the exact right set of definitions to use, at which point proofs tend to pop out as trivial and obvious statements using those definitions. And at a more practical level, you won't be able to read any math if the definitions are not ingrained, so you might as well get a head start and just rote memorize them if you want to succeed.

But it's interesting that the language is far less sticky in memory than the underlying intuition. My guess is that because the intuition is so much harder to develop, it wires itself in much more deeply than the words themselves, which can be pretty easily learned in a few hours of flashcard work.

The entire p-zombie concept presumes dualism (or something close enough to it that I'm happy to lump it all into the generic category of "requires woo"). Gravity having an effect on something is measurable and provable, whereas qualia are not.

We don't know every fact, either, so I don't know how you can use that idea to say that we're not Turing machines. Apart, of course, from the trivial fact that we are far more limited than a Turing machine...

Adult humans can do symbolic reasoning, but lower mammals cannot. Even ones that share most of our brain structure are much worse at this, if they can do it at all; children need to learn it, along with a lot of the other things that we consider a natural part of human intelligence.

That all points towards symbolic reasoning being a pretty small algorithmic discovery compared to the general ability to pattern match and do fuzzy lookups, transformations, and retrievals against a memory bank. It's not like our architecture is so special that we burned most of our evolutionary history selecting for these abilities, they're very recent innovations, and thus must be relatively simple, given the existence of the core set of abilities that our close ancestors have.

The thing about transformers is that obviously they're not the end of the line, there are some things they really can't do in their current form (though it's a smaller set than people tend to think, which is why the Gary Marcuses of the world always backpedal like crazy and retcon their previous statements as each new release does things that they previously said were impossible). But they are a proof of concept showing that just about the simplest architecture that you could propose that might be able to generate language in a reasonable way (beyond N-gram sampling) can, in fact, do it really, really well even if all you do is scale it up, and even the simplest next-token prediction as a goal leads to much higher level abilities than you would expect. That was the hard core of the problem, building a flexible pattern mimic that can be easily trained, and it turns out to get us way further along the line to AGI than I suspect anyone working on it ever expected it would without major additions and changes to the design. Now it's probably time to start adding bits and bobs and addressing some of the shortcomings (e.g. static nature of the network, lack of online learning, the fact that chains of thought shouldn't be constrained to token sequences, addressing tokenization itself, etc), but IMO the engine at the heart of the current systems is so impressively capable that the remaining work is going to be less of an Einstein moment and more of an elbow grease and engineering grind.

We may not be close in the "2 years of known work" sense, but we're certainly not far in the "we have no idea how to prove the Riemann Hypothesis" sense anymore, where major unknown breakthroughs are still required which might be 50+ years away, or the problem might even be unsolvable.

Another view is that the Internet exactly matched this model, and that it's much closer to the norm than not: the Internet became available to normal people in the mid-to-late 90s, depending on where you were, but all that was on the web was dumb personal websites (1). In the late 90s bubble, startups seemed pretty crazy, slapping "...but on the Internet" onto basically any idea and raising money off of it (2), basically what's happening with "...but with AI" today. Most failed to find a market because too few people were using the Internet for everything and the frantic search for users failed (3).

But just as the conservative old-school business people were laughing and patting themselves on the back post-bubble over how stupid all the dotcoms were for thinking they could monetize eyeballs, Google emerges, and 20 years later tech companies drive the stock market rather than following it. Don't dismiss a technology just because the birthing spasms look ugly, it takes some time for markets to develop and for products to settle into niches. At the start a lot of that is due to people not being comfortable, tech sucking, and the market shifting too quickly to precisely target, but that can all change pretty fast.

If you roll it back upon seeing problems, then you're doing something meaningful, at least. IMO 90+% of the value of A/B testing comes from two things, a) forcing engineers to build everything behind flags, and b) making sure features don't crater your metrics before freezing them in and making them much more difficult to remove (both politically and technically).

Re: b), if you've ever gotten into a screaming match with a game designer angry over the removal of their pet feature, you will really appreciate the political cover that having numbers provides...

No, they just ship their org chart. Which has grown incredibly, to the point where even internally nobody knows what's being shipped three branches over on the tree.

So, what's happening here on the surface is that it's an optimization (fairly meaningful, from the looks of it) aimed at doing roughly the same things we could already do with chain-of-thought (CoT), but IMO the downstream effects of this sort of optimization could be much more meaningful.

LLMs can already do a decent amount of "processing" in a single token generation because of the number of layers they have. The layers are trained independently so it's not exactly like they're a recurrent network doing multiple steps, but they are layering sequences of context-dependent transformations on top of each other; no matter how you cut it, if getting to a problem's answer requires 100 steps, you won't be able to do it in a single token output from a 20 layer LLM. To some approximation, CoT is just a way to give the network more chances to transform the data than there are layers in the network - each additional token of output gives a shot to bake another vector the size of the token embedding into each layer's state in the network, enriching what it's computed so far.

The problem with chain of thought is that as you add each new token, at the input level of the network, your computation is basically starting from scratch against the raw text, just with one additional token. You don't even have access to all the stuff you already figured out in the deepest layers of the network during the previous step! If you were processing "All wunguses are glurgles, and Joe is a wungus", then somewhere in those deepest layers as you're generating the next token you've almost certainly got some vector that basically represents "therefore Joe is a glurgle", but with chain of thought you've got to first output "t", then "h", then "e", and so on (I know those aren't tokens, let's pretend letter == token for argument sake), and during that process almost ALL of the work being done by the network is mere bookkeeping, slowly dumping that thought into the output stream. Only once you get the whole sentence out can you start processing the next token at the first layer with the information that Joe is, in fact, a glurgle, in hand. Which is a damn shame, because it's been sitting right there in the deeper layers of the network parallel to previous tokens this whole time, it just wasn't available for the shallow layers to process directly because you were casting most of the info away and "rounding" to a single token.

With Coconut's approach, you don't need to output "therefore Joe is a glurgle" token by token to continue the train of thought, you can essentially pass the entire thought through as a single uber-token, and the next pass can generate a new entire thought, and so on.

It's a pretty straightforward idea, IMO the neat bit is that they were able to train the network to work well in this way by leveraging CoT. I'm guessing you probably don't need to act as if these are two distinct modes of operation, you could instead always have this side channel of "continuous thought" running, even when you have generated a normal token, coming through as a separate input to the first attention block. You still might want to have a "thinking" token when you need to sit there and let the thing do more work, but you'd generally increase the information flow from time step to time step, which would allow the net to keep thinking in the background even as it's doing the gruntwork of outputting whatever its current "buffered" thought is.

Counterpoint: our CEO spent 25 minutes shitting out a bunch of AI ads because he was frustrated with the pace of our advertising creative team. They hated the ads that he created, for the reasons you mention, but we tested them anyways and the best performing ones beat all of our "expert" team's best ads by a healthy margin (on all the metrics we care about, from CTR to IPM and downstream stuff like retention and RoAS).

Maybe we're in a honeymoon period where your average user hasn't gotten annoyed by all the slop out there and they will soon, but at least for now, there is real value here. Yes, out of 20 ads maybe only 2 outperform the manually created ones, but if I can create those 20 with a couple hundred bucks in GenAI credits and maybe an hour or two of video editing that process wipes the floor with the competition, which is several thousand dollars per ad, most of which are terrible and end up thrown away, too. With the way the platforms function now, ad creative is quickly becoming a volume-driven "throw it at the wall and see what sticks" game, and AI is great for that.

I don't think that's really true in practice, though at a competence level it absolutely is. Except at the biggest places, even low level programmers are both empowered and expected to make architectural decisions that have big impact on maintainability, testability, and performance. Addition of a bad-but-fast coder to a team is the worst possible outcome because they are really productive at screwing up the whole codebase in ways that will take even more time to peel apart. It's actually worse if their code is in the "functional but ugly" category, since PMs love those people as long as they're really fast, and they hate when other engineers complain about their work.

In construction, the random guy pulled in to swing a hammer may cut something wrong once or twice and waste a little time and money, but he's never going to be trusted to design the way the support beams hold up the whole house, so the damage is very limited. Software is not like that, we expect everyone to do some amount of design and engineering, whether or not they have any ability to do so. If as much software dev was driven by very low-level grunt work as is the case in construction, LLMs would already have revolutionized the field a lot more than they have; as it is, we've probably got another couple years to wait.