sigbottle

But a property of intelligence is to know when to stop, if we treat intelligence as some sort of search and not some a priori intuition of the entire space. Seems kind of hard, if not impossible, to train for specifically that.

Really? Not my experience at all.

If you're saying that math has a consolidating network effect, sure. There are a lot of people who think similar problems are important. I guarantee you that when math was small, that is not the case, and when coming up with new math, that's not the case, and when coming up with sub problems to tackle large problems, that is not the case.

Completely unrelated, but I'm seeing people and especially LLMs using causal/intervention so much it's kind of driving me insane.

It's actually a very goated term but not everything is causal, it also has precise technical meanings (although those get blurred too given that causal can mean anything from intervention proper, to mere depdnence on something prior)

Ah, I see. We don't care at that level because we aren't trying to reproduce deterministic replicated simulations as the HFT industry often wants. SDRs have their own noise and the physical world is uncertain to where, as long as your algorithm isn't above, yeah say the 10^-5 to 10^-8 range in output, it's probably fine.

It's probably the determinism and reproducibility that's the key constraint in HFT settings. In other industries it's just generic textbook numerical analysis (analyze batched eps error), and then in engineering land it's "ehh good enough".

Hmm - I think you mean to say that when interoperability is not a core constraint, then there won't be an incentive for an open standard.

That's important because in plenty of other software - databases, OSes, compilers and binary formats, networks, etc. name your favorite CS topic, there is an open standard for it.

You can either achieve interoperability by... achieving interoperability, or by dominating market share and edging the user's action space to solely stay in your ecosystem (change the action distribution of your user) that for all intents and purposes that it "interoperates with everything" means that, "since I bought entirely into this ecosystem, I interoperate with everything".

So ultimately I agree with you, but to present it as a failure of objective engineering prowess on open source is weird. As you said yourself, it's clearly the network effects that prop the business model. And sure, gaining market share isn't a trivial task, but it's not engineering.

I think framing it in the interoperability way is healthier - because it actually provides a potential new out to work towards, rather than this weird beration double-bind that frames Microsoft's approach as The Objective Way to do things. It's not and doesn't have to be.

I've used LLMs to bootstrap successfully in a decent amount of things at this point.

Anyone trusting AI as the single authoritative source of information is stupid - but this follows from the fact that trusting anyone as a "singular point" as a source of information is stupid. You corroborate, you intervene on the world to test your mental model, you discuss with other people. That's what learning is. I've never learned from start to back to a textbook before as the single source of information (besides one philosophy of science textbook; in which I spent a month digging around adjacent fields, and then it just so happened that that one textbook synthesized every piece of information I looked up, and it was mostly a consolidating review).

If your study pre-supposes certain courses of action and artificially constrains the action space for the sake of "reproducibility", you may get a result, and a "scientifically rigorous one". But it's not going to say anything about reality in any meaningful way. While anecdotes and the complexity of real life isn't "science" (in that it's a controlled, repeatable, interventional experiment that's subject to a community of critics who want to hold you up to standards of rigor), there's far more truth in how people actually proceed and engage with these tools.

For physics simulations, or even in RF work, the tolerances you need are quite a bit more than say, AI work. For example, for one of our RF algorithms, we "have" to use doubles (I actually can think of an optimization that can let us use floats, but it's quite involved to fix it) to get the required precision. That precision requirement is 10^-15. For physics simulations, I've heard of accuracies of up to 10^-20.

I will say, generally, it depends on what you mean. There's like, numerical drift, and then there's converting to the discrete part. Generally, I'd say that "getting into the discrete part" of an algorithm, unless you're trying to match behavior of another piece of software, is fine. Like, if you're playing a game, people are not going to care if the shot just grazed them and missed, but the "Actual True Value" is like 10^-9 off or something. They will just chalk it up to "close, unlucky". If you have one canonical place for your floating point errors from which your "discrete" parts derive, it's no biggie.

But at work, I recently debugged an algorithm in which the author, effectively wanted to compute the integer and fractional parts of a double type. Their reasoning was that, for the fractional part, there can be imprecision, so you should do some fancy math to recover the fractional part.

This is giga wrong for the application we were building. You never want to try computing the "same double" in two different ways and compose the results like this. Intuitively, if one path "thinks" your double is 15.00001, and the other path "thinks" its 14.99999, you're now saying that the double is 15 (integer) + .99999 (fractional) = 15.99999: this is way off now!

You should just use one double, and extract out the integer and fractional parts. It doesn't matter if the double is imprecise - it's authoritatively imprecise. Much easier to reason about.

This is the first time I've debugged an edge case like this since I wouldn't implement something like this, but it's an example of how subtle floating point errors can be.

The standards of proof are different from the fundamental operation of "OK, cool, you solved this problem. Why does this problem matter? Isn't it useless? Senseless? Meaningless?" You have this same question whether or not you're in an a priori discipline (mathematics), scientific fields proper, or engineering. "Absolute certainty" has nothing to do with it. I can assure you, people on the job are not looking for The Absolute Truth when doing their jobs, yet they still can question at a solution by asking: are we solving the right problem?

(Although in general, there's no true difference between "I answered the question correctly, but the question was mapped to this thing we call 'reality' wrong", and "I answered the question incorrectly", because you can (try) adding the constraints that you really wanted targeted in case A, to case B, and boom, suddenly a question/answer pair that was "Answered correctly, but question doesn't map to reality" now becomes, "You answered this question wrong". However, individuals generally tend to have some breakpoint to differentiate between the two).

I'm going to keep on repeating this on HN threads until I'm blue in the face, but:

There are two ways to solve a problem. Either solve the problem, or deem it irrelevant.

The implication here is that, you, the human operator, clearly are just confused. The LLM knows best. You're just a stupid human. The LLM knows objective truth, you do not. You have concerns, questions, the LLM didn't understand your question "properly"? Do not worry, the LLM objectively knows the optimal course of action. It thought through the implications of what you said, took into account all possible data, and came to the objectively correct design for your software, your society, your life.

In some sense, this problem would have been a societal problem within the next several decades anyways, but it's been hyper-accelerated by AI.

A lot of philosophy starts from the fundamental observation that, to solve a problem, you could either solve a problem, or state that the problem is ill-posed in some way (and dissolve it). Either answer the question, or question the question.

It's not a new problem in some sense. If you've dealt with really smart but really arrogant friends, they might jump ahead 10 steps and assume your rebuttal, posit theirs, assume your rebuttal to their posit, etc. etc. without... actually taking the time to listen carefully. On the national scale, this looks like forced trust in government authorities about what is "objectively best".

People need to get it through their skulls that, even if an AI, or any intelligence, could even solve the damn Riemann Hypothesis: if it's wrong, it's wrong. Of course, I think all of us know the objection - we see it on hackernews all the time. "You guys are just stupid contrarians who can't understand AI's deep reasoning". OK. The second inference? "therefore you are unable to govern yourselves properly - your concerns are all fallacies, misunderstandings, bad for you, etc.".

You might think that the second inference is extreme and nobody actually believes that, but as always, it's a gradient. Before AI, you might've had an extremely strong sense of self. Now? You look at OpenAI solving open math problems left and right on a public foundation model, and you think, "Maybe I should just trust it more. If I spend cycles thinking, it's probably going to outdo me anyways." The AI model silently makes 5 different assumptions and transformations? "Well, maybe it was rational in the space of tradeoffs to do that. The AI knows best, after all". You might be thinking of an architecture with 5 different key constraints based on lived experience, in which the AI keeps misunderstanding. "Oh, well, this genius-level mathematician/programmer AI isn't understanding my words - surely I must be mistaken, right? It's only humble to think that way".

I can't convince people otherwise though. After all, I can't "prove" that you should have a backbone when talking to AI. it could just as easily be "you're arrogant, this machine is in the top of all academic fields and is coming for all white collar jobs, who's to say you're right about anything?" All I can say is, there's a reason why Dostoevsky is one of my favorite authors.

A lot of the AI game is still integrations and getting entrenched in processes. It has the double whammy that you get to train on their data (not talking about ALL enterprise contracts - but cursor pretty explicitly has this business model. So I'd imagine some enterprise contracts have this.)

I have a sinking feeling that AI models have gotten so good internally, and/or the GPU guys in their labs have found a good enough processes, that AI is doing the research behind the doors right now (it should be at least capable, at heuristic optimization problems that are benchmarkable), while the hard part of their business is the political game: convincing other companies to buy into AI, integrations, trying to gather more data where they can.

The "singularity" as stated requires AI to either make a technological advancement strong enough to be deadly to humans (besides just intelligence), or spreading deep institutional support for itself among society.

The idea that "we will get superintelligence first, then... ???" is kind of a weird notion. I mean, it's pretty arguable that we do have at least some form of superintelligence. The AI itself needs to actually do something with it though. Either that, or more likely, someone needs to do something bad with the superintelligence.

That could be both re-assuring or not. Because under that view, given how AI is being integrated so quickly into society, it's not going to take this fantasy view of superintelligence to reach the singularity. If you have broad institutional support and crowd out the thing we call humanity over time (the two ways to 'solve' a problem: solve it, or declare it meaningless), that is another way to reach the singularity.

do world cup athletes get a big bonus for world cup participation from their govts and/or FIFA?

Bertrand Russell objected to the notion of causation in the 1900s, because merely stating the updated dynamics of a system doesn't imply causation in any grand sense. Like hume, he dismisses causation, but not because of the problem of induction or anything, but because the concept seems incoherent to him. He especially emphasized this in physics - although maybe you can argue that for everyday human language, causation is good (Alice caused BCD to happen), in physics it doesn't belong.

Not that I entirely agree with his account but just some food for thought.

But you can just apply this to anything. I feel unlike unless you're an insider with skin in the game, your criticism doesn't land other than a generic surface one.

To be clear, I apply an equal deep skepticism to most fields that aren't math (in the sense of a priority) or physics (in the sense that you aren't trying to study the entire world, but a specific set of phenomena that you can reliably control enough + repeat to run intervention on), whether the results agree with me or not. Maybe a bit of intellectual closed-mindedness. But then that means that me, personally, I can't in good faith use the criticism as a proper 'debunk' argument - at best it's a heuristic to avoid spending cycles to evaluate it (which is 'rational' behavior, as much as I hate that word, IMO).

If humans are still around by the end of the century: previously shunned philosophical movements such as pragmatism will actually become mainstream again.

The most interesting problems arise when you don't try and force one shared standard upon everybody yet still try and play nice.

Alternatively, power could concentrate and the winners get to decide what is valuable and not, thus cutting down the space of possible complexity by construction.

What's the hiring space and business strategy around all of these smaller AI labs? Its really cool that people like these guys get paid to optimize models and give them out for free (open source). Do a lot of these labs have forward deployed engineers doing integrations with customers who want local models? Is there a general shift towards the local model crowd?

I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all.

Something like this is actually a stance in the tradition of inferentialism (see the term sapience). Though "reasoning" isn't like, turing machine computability in this space; from what I understand, it's some abstract notion of the "space of reasons". I don't really understand it, honestly.

There's some merit to this, IMO. When an LLM goes wrong, do you blame the person or the LLM? As in, would you throw said LLM in jail, and hold the LLM accountable? Not right now, at least. I'm not sure if that's what is meant by the "space of reasons", but the intuition is that 'reason' can mean a lot of different things, pragmatically speaking. Reason as a legible audit trail is one of those ways.

But that's arguably getting into the social aspect of 'reason' (important!) and not like, what STEM people traditionally think of as 'reason'.

If you want to take it that far, we've had results like this since the 60s (solomonoff induction). But of course if you state it like that, your (rightful) objection is that it's pretty vacuous (if I had the computational omniscience to just brute force all possible turing machines, whatever that even means, then sure, any 'f' gets subsumed into this paradigm).

A lot of philosophers, mathematicians, scientists, etc. effectively say, "Yeah, everything's just f(inputs of world) -> outputs!" That 'f' is doing a lot of heavy lifting. Which is kind of the point of mechanistic interperability - to make sure we're not jumping ahead of ourselves, and to make sure we're careful when we claim what "deep" and "structure" means, when it pertains to that 'f'.

Is the implication that human thinking is inherently inferior to the perfect, all knowing AI?

Again, I find this line of thinking time and time again in the modern AI booster space. There are two ways to deal with a problem. Either deal with it, or make it not a problem. Yes, if everyone was simply AI, maybe there would be no problems, because there's no "problematic thought distributions", but that's not how the world is, is it?

And I suspect that even in your hypothetical, perfect rational world, agents would have "cognitive dissonance and contradictory preferences."

And even in this case, even aside from the inherent complexities in a coherent account of thinking and rationality, what the fuck? Not uploading your entire user home directory is clearly within the rules of a hypothetical non-malicious, intelligent AI. Just because an account of all thought is hard, doesn't mean that some thoughts aren't cut and clear.

How was integration for you guys? Was the integration easier for you guys since you have an established emacs system consuming terminal output with reasoned semantics over what goes where for existing subsystems (rendering, osc codes, etc)?

For libghostty-vt, since you're targeting a terminal TUI instead of an external subsystem (for example; for ghostty, you hit libghostty-vt -> GPU rendering, which is external), you still have to buy into terminal semantics. in my experience, since I was trying to replicate mosh with libghostty-vt as the parser, what happened was that my optimized re-rendering kept getting increasingly coupled to terminal semantics (and the UDP state update model too), otherwise I'd have to send the entire terminal grid over the network like, every time.

What are the tricks for making this both performant and not like, utter cancer? You have a harder issue here too (similar to tmux) in that certain optimizations are just not available to you, or you have to translate (literally geometrically) certain instructions

Auto-vectorization is not a programming model!

There was no useful math after 1964? What? Or do you not count the entire field of computer science as math (arguably FFT belongs to computer science too)?

what kinds of proofs would it be good at? I thought that combinatorial proofs would be easier to reason over than ones that required analysis

I mean, people always say there are tradeoffs, until you reach the next frontier, in which there are tradeoffs at said frontier, and the next, and the next, etc.

In one sense, yes, tradeoffs are inescapable as the scope expands to the maximal possible scope. In another sense... it depends on the level of abstraction we're talking about.

I highly respect many people at cognition but yeah that's put a sour taste in my mouth.

I want to work in the AI space on actual AI research, at any part of the stack. Even if I'm developing training infra - as long as people are advancing knowledge of what intelligence could be.

But it seems like either it's big labs or grifters, that's it, and even the big labs, at least publicly, seem very grifty at times. Not like I have the technical chops probably, but still.

From an interoperability perspective, this breaks the advantage of LLM inference that frontier AI labs have, in that you just have everyone run through the same algorithm but configure via text.

If you added probes at the model layer, you have to serve multiple different types of kernels at the same time, for multiple different companies and use cases (I guess you could provide a standardized set of probes for users), start tracking version control for each of the kernels, etc. very nasty compared to right now.

Could be a really interesting problem in the next 10 years or so, but this would require labs to be far more open about their models; and labs are still shooting for their AGI anyways, with the idea that nothing you suggest right now matters if AGI exists in a decade.

I mean mathematicians have encoded a decent amount of the operations you mean for messy versus true / false. Turing machines are built from mere Boolean logic, but also notions of compression, information theory, projections, function mappings, etc. provide all the math-crank language you want to reformulate knowledge work.

The issue is that mathematics really does not have a good answer for how things update.

When I disagree with the data: I will nitpick every last detail of methodology, any cross-corroboration is an anecdote, suddenly I demand a-priori levels of justification. All science is flawed anyways, it's not like mathematics, you can't get absolute certainty, so why bother? You're always going to be making base assumptions that can be challenged, you're necessarily going to abstract out the territory, the map is flawed.

When I agree with the data: I will boast about the victories of science and empiricism, we found the perfect set of natural abstractions that are necessary and sufficient to map out the territory that carve at the joints of the problem, any concern about assumptions is rebutted with generic "Well, we're just pragmatists; we're not perfect, but clearly we're converging on the right direction! You're clearly someone who just wants to nitpick and not get any work done."

My experience with certain hackernews commenters in a nutshell.