lechatonnoir

I mean... you can't think of any ways that AI could actually generate new value? Or more abstractly, of a way that Jevons' paradox can't apply in the case of AI?

i think by your logic, they only thing that they do that is condescending is to say that an interview is not guaranteed.

people are mentioning that they do this for a reason, which explains away that behavior, so yeah, it kinda does change the fact of whether they are being condescending.

it's kind of hard to tell what your position is here. should people not ask chatbots how to scrape html? should people not purchase RAM to run chatbots locally?

i think in this thread the goalposts were slowly moved. people were initially talking about success being predicted by having the excess necessary to comfortably take many shots on goal. it seems like we've granted that this $250k shot was a one-time thing.

it is true but irrelevant to the original topic that this is more money than the global poor ever see, and that this is more money that most people get to have. i don't think anyone was arguing that this represents zero privilege

I'm pretty sure there's no reason that Anthropic has to do research on open models, it's just that they produced their result on open models so that you can reproduce their result on open models without having access to theirs.

Hmm, that might be possible, but that's essentially not what I assumed. At the very least, they operate on the same hardware, so Autopilot is in some sense "intentionally bad" as a whole.

I am sure that there are some people who exhibit the behaviors you're describing, but I really don't think the group as a whole is disinterested in prior work or discussion of philosophy in general:

https://www.lesswrong.com/w/epistemology

https://www.lesswrong.com/w/priors

https://www.lesswrong.com/posts/2x67s6u8oAitNKF73/ (a post noting that the foundational problems in mech interp are grounded in philosophical questions about representation ~150 years old)

https://www.lesswrong.com/w/consciousness (the page on consciousness first citing the MIT and Stanford encyclopedias, then providing a timeline from Democritus, through Descartes, Hobbes,... all the way to Nagel, Chalmers, Tegmark).

There is also sort of a meme of interest in Thomas Kuhn: https://www.lesswrong.com/posts/HcjL8ydHxPezj6wrt/book-revie...

See also these attempts to refer and collate prior literature: https://www.lesswrong.com/posts/qc7P2NwfxQMC3hdgm/rationalis...

https://www.lesswrong.com/posts/xg3hXCYQPJkwHyik2/the-best-t...

https://www.lesswrong.com/posts/SXJGSPeQWbACveJhs/the-best-t...

https://www.lesswrong.com/posts/HLJMyd4ncE3kvjwhe/the-best-r...

https://www.lesswrong.com/posts/bMmD5qNFKRqKBJnKw/rigorous-p...

Now, one may disagree with the particular choices or philosophical positions taken, but it's pretty hard to say these people are ignorant or not trying to be informed about what prior thinkers have done, especially compared to any particular reference culture, except maybe academics.

As for the thing about Aella, I feel she's not as much of a thought leader as you've surmised, and I think doesn't claim to be. My personal view is that she does some interesting semi-rigorous surveying that is unlikely to be done elsewhere. She's not a scientist/statistician or a total revolutionary but her stuff is not devoid of informational value either. Some of her claims are hedged adequately, some of them are hedged a bit inadequately. You might have encountered some particularly (irrationally?) ardent fans.

Here's a collection of debates about that topic:

https://www.lesswrong.com/posts/85mfawamKdxzzaPeK/any-good-c...

I personally don't have that much of an interest in this topic, so I can't critique them for quality myself, but they may at least be of relevance to you.

I am really not sure where you get any of these ideas. For each of your critiques, there are not only discussions, but taxonomies of compendiums of discussions about the topics at hand on LessWrong, which can easily be found by Googling any keyword or phrase in your comment.

On "considering what should be the baseline assumption":

https://www.lesswrong.com/w/epistemology

https://www.lesswrong.com/w/priors, particularly https://www.lesswrong.com/posts/hNqte2p48nqKux3wS/trapped-pr...

On the idea that "rationalists think that they can just apply rationality infinitely to everything":

https://www.lesswrong.com/w/bounded-rationality

On the critique that rationalists are blind to the fact that "reason isn't the only thing that's important", generously reworded as "reason has to be grounded in a set of human values", some of the most philosophically coherent stuff I see on the internet is from LW:

https://www.lesswrong.com/w/metaethics-sequence

https://www.lesswrong.com/w/human-values

On "systematically plan to validate":

https://www.lesswrong.com/w/rationality-verification

https://www.lesswrong.com/w/making-beliefs-pay-rent

On "what could hold true for one moment could easily shift":

https://www.lesswrong.com/w/black-swans

https://www.lesswrong.com/w/distributional-shifts

https://www.lesswrong.com/w/forecasting-and-prediction

Well, yeah, I think it's a pretty socially unaware thing to say about yourself out loud, so that's a pretty strong filter there.

It's rather different for a community to say that's a standard they aspire to, which is a lot less ridiculously grandstanding of a position IMO.

I suppose you're right about that, so I can't make the argument go through by saying "mathematics" vs "philosophy". Maybe what I should say instead is that as some dialectics advance/technologies develop, subfields of both such things sprout up and have a lot of low-hanging fruit to pick, and in these cases, the new work will be descended from but not too-essentially informed by the prior work.

Like mathematical logic (in the intersection of math and philosophy) didn't have that many true predecessors and was developed very far by maybe only 5-10 individuals cumulatively, or information theory was basically established by Claude Shannon and maybe two other guys, or various aspects of convex optimization or Fourier analysis were only developed in the 80s or so, it stands to reason that the AI-related applications of various aspects of philosophy are ripe to be developed now. (By contrast, we don't see, as much, people on LW trying to redo linear algebra from the ground up, nor more "mature" aspects of philosophy.)

(If anything, I think it's more feasible than ever before, also, for a bunch of relative amateurs to non-professionally make real intellectual contributions, like noticeably moreso than 100 or even 20 years ago. That's what increasing the baseline levels of education/wealth/exposure to information was intended to achieve, on some level, isn't it?)

I think you're conflating different groups of people pretty severely.

"shunned" in particular is a really strong word, e.g, global health and biosecurity are two of the named categories at the most central EA events:

https://www.effectivealtruism.org/ea-global/events/ea-global...

Well, on a meta level, I think their community has decided that in general it's better to post (and subsequently be able to discuss) ideas that one is not yet very confident about, and ideally that's what the "epistemic status" markers are supposed to indicate to the reader.

They can't really be blamed for the fact that others go on to take the ideas more seriously than they intended.

(If anything, I think that at least in person, most rationalists are far less confident and far less persuasive than the typical person in proportion to the amount of knowledge/expertise/effort they have on a given topic, particularly in a professional setting, and they would all be well-served to do at least a normal human amount of "write and explain persuasively rather than as a mechanical report of the facts as you see them".)

(Also, with all communities there will be the more serious and dedicated core of the people, and then those who sort of cargo-cult or who defer much, or at least some, of their thinking to members with more status. This is sort of unavoidable on multiple levels-- for one, it's quite a reasonable thing to do with the amount of information out there, and for another, communities are always comprised of people with varying levels of seriousness, sincere people and grifters, careful thinkers and less careful thinkers, etc. (see mobs-geeks-sociopaths))

(Obviously even with these caveats there are exceptions to this statement, because society is complex and something about propaganda and consequentialism.)

Alternately, I wonder if you think there might be a better way of "writing unconfidently", like, other than not writing at all.

Aside from the remark given in the other reply to your comment, I wonder what the standard is: how quickly should a community appear to correct its incorrect beliefs for them to not count as sheep?

They don't really ever show a sense of "hey, I've got a thought, maybe I haven't considered all angles to it, maybe I'm wrong - but here it is". The type of people that would be embarrassed to not have an opinion on a topic or say "I don't know"

edit: my apologies, that was someone else in the thread. I do feel like between the two comments though there is a "damned if you do, damned if you don't". (The original quote above I found absurd upon reading it.)

Well, sure, but mathematics is the domain for which this holds maybe the most true out of any. It's less true for fields which are not as old.

I'm not sure if this counterpoint generalizes entirely to the original critique, since certainly LessWrongers aren't usually posting about or discussing math as if they've discovered it-- usually substantially more niche topics.

This is such a pointless, tired take.

You want to say this guy's experience isn't reproducible? That's one thing, but that's probably not the case unless you're assuming they're pretty stupid themselves.

You want to say that it Is reproducible, but that "that doesn't mean AI can think"? Okay, but that's not what the thread was about.

I think the idea is, why would the visualization be so intentionally bad in the Autopilot version as to not detect the kids entirely? What benefit does that confer, or, from another perspective, what software constraint forces this to be the case?

I would like to understand how you ideally imagine a person solving issues of this type. I'm for understanding things instead of hacking at them in general, and this tendency increases the more central the things to understand are to the things you like to do. However, it's a point of common agreement that just in the domain of computer-related tech, there is far more to learn than a person can possibly know in a lifetime, and so we all have to make choices about which ones we want to dive into.

I do not expect to go through the process I just described for more than a few hours a year, so I don't think the net loss to my time is huge. I think that the most relevant counterfactual scenario is that I don't learn anything about how these things work at all, and I cope with my problem being unfixed. I don't think this is unusual behavior, to the degree that it's I think a common point of humor among Linux users: https://xkcd.com/963/ https://xkcd.com/456/

This is not to mention issues that are structurally similar (in the sense that search is expensive but verification is cheap, and the issue is generally esoteric so there are reduced returns to learning) but don't necessarily have anything to do with the Linux kernel: https://github.com/electron/electron/issues/42611

I wonder if you're arguing against a strawman that thinks that it's not necessary to learn anything about the basic design/concepts of operating systems at all. I think knowledge of it is fractally deep and you could run into esoterica you don't care about at any level, and as others in the thread have noted, at the very least when you are in the weeds with a problem the LLM can often (not always) be better documentation than the documentation. (Also, I actually think that some engineers do on a practical level need to know extremely little about these things and more power to them, the abstraction is working for them.)

Holding what you learn constant, it's nice to have control about in what order things force you to learn them. Yak-shaving is a phenomenon common enough that we have a term for it, and I don't know that it's virtuous to know how to shave a yak in-depth (or to the extent that it is, some days you are just trying to do something else).

I relate to this a bit, and on a meta level I think the only way out is through. I'm trying to embrace optimizing the big picture process for my enjoyment and for positive and long-term effective mental states, which does include thinking about when not to use the thing and being thoughtful about exactly when to lean on it.

I basically agree, but with the caveat that the tradeoff is the opposite for a bunch of tedious things that I don't want to invest time into getting better at, or which maybe I only do rarely.

Well said, these things are actually in a tradeoff with each other. I feel like a lot of people somehow imagine that you could have the best of both, which is incoherent short of mind-reading + already having clear ideas in the first place.

But thankfully we do have feedback/interactiveness to get around the downsides.

Well said about the fact that they can't introspect, and I agree with your tip about starting with fresh context, and about when to give up.

I feel like this thread is full of strawmen from people who want to come up with reasons they shouldn't try to use this tool for what it's good at, and figure out ways to deal with the failure cases.

Well, in that case, the LLM company has made a mistake in marketing their product, but that's not the same as the question of whether the product works.

I somehow can't reply to your child comment.

It depends on whether the cost of search or of verification dominates. When searching for common consumer products, yeah, this isn't likely to help much, and in a sense the scales are tipped against the AI for this application.

But if search is hard and verification is easy, even a faulty faster search is great.

I've run into a lot of instances with Linux where some minor, low level thing has broken and all of the stackexchange suggestions you can find in two hours don't work and you don't have seven hours to learn about the Linux kernel and its various services and their various conventions in order to get your screen resolutions correct, so you just give up.

Being in a debug loop in the most naive way with Claude, where it just tells you what to try and you report the feedback and direct it when it tunnel visions on irrelevant things, has solved many such instances of this hopelessness for me in the last few years.

One question is whether, even after all that backpedaling, you feel you could've achieved the same or a similar result in those five days. My findings have been that it's a net plus for productivity, but I'm a bit less sure whether I prefer the way work feels when a lot of it is just going back and cleaning up after the growth. (Of course, that sounds like a familiar statement for a lot of engineers before LLMs, too.)

Well, the first thing they said was that it at least removed the need for their commute, which might be something. In general, it does take resources to create the conditions for people to work. Maybe there will be room for new value for the existing people as a result.

I am mocking you, but you didn't get it.

you assert that people know a killer app when they see one

a bunch of people think that something like chatgpt is a killer app, and they know it when they see it. you assert that it obviously is not, so clearly the above intuition isn't working for the purposes of discussion.

instead, someone should define the term so that we know what we're talking about, and i offer you the ability to do it so that the frame of the discussion can be favorable to your point of view. but you are also not willing to do that, so how do you expect to convince anyone of your viewpoint?

"gdm pioneered rl" is definitely not actually right, but it's correct to assert that they were huge players.

people who knew from context that your statement was broadly not actually right would know what you mean and agree on vibes. people who didn't could reasonably be misled, i think.