Does current AI represent a dead end?

https://www.bcs.org/articles-opinion-and-research/does-current-ai-represent-a-dead-end/

by jnord • 2 years ago

543 1034 2 years ago

bcs.org

goodpoint• 2 years ago

Yes.

crest• 2 years ago

Only if you care about about causation instead of just correlation.

optimalsolver2• 2 years ago

Yes.

An actual "thinking machine" would be constantly running computations on its accumulated experience in order to improve its future output and/or further compress its sensory history.

An LLM is doing exactly nothing while waiting for the next prompt.

xvector4• 2 years ago

Is a human with short term memory loss - or otherwise unable to improve their skills - generally intelligent?

rcarmo2• 2 years ago

A human with short term memory loss still has agency and impatience.

optimalsolver2• 2 years ago

Would you let such a person handle an important task for you?

aetherson• 2 years ago

There is a limited amount of computation that you can useful do in the absence of new input (like an LLM between prompts). If you do as much computation as you usefully can (with your current algorithmic limits) in a burst immediately when you receive a prompt, output, and then go into a sleep state, that seems obviously better than receive a prompt, output, and then do some of the computation that you can usefully do after your output.

amelius• 2 years ago

Can't we just finetune the model based on the LLM's output? Has anyone tried it?

soulofmischief• 2 years ago

Not only does a training pass take more time and memory than an inference pass, but if you remember the Microsoft Tay incident, it should be self-explainatory why this is a bad idea without a new architecture.

m3kw9• 2 years ago

We are thinking machines and we keep thinking because we have one goal which is to survive, machines have no such true goals. I mean true because our biology forces us to do that

alchemist1e9• 2 years ago

self prompting via chain of thought and tree of thought can be used in combination with updating memory containing knowledge graphs combined with cognitive architectures like SOAR and continuous external new information and sensory data … with LLM at the heart of that system and it will exactly be a “thinking machine”. The problem is currently it’s very expensive to be continuously running inference full time and all the engineering around memory storage, like RAG patterns, and the cognitive architecture design is all a work in progress. It’s coming soon though.

whatwhaaaaat• 2 years ago

We’re going to need to see this working. From my perspective many of the corporate llms are actually getting worse. Slop feedback loops.

By no means has it been proven that llms functioning the way you describe will result in superior output.

Uehreka2• 2 years ago

I see people say this all the time and it sounds like a pretty cosmetic distinction. Like, you could wire up an LLM to a systemd service or cron job and then it wouldn’t be “waiting”, it could be constantly processing new inputs. And some of the more advanced models already have ways of compressing the older parts of their context window to achieve extremely long context lengths.

Earw0rm4• 2 years ago

If it's coalescing learning in realtime across all user/sessions, that's more constant than you're maybe giving it credit for. I'm not sure if GPT4o and friends are actually built that way though.

wat100003• 2 years ago

If you had a magic stasis tube that kept a person suspended while you weren’t talking to them, they’d still be a thinking machine.

rcarmo2• 2 years ago

Yes. Next question, please. And don't mention AGI.

xvector4• 2 years ago

IMO we are already at AGI. Hell, Norvig would argue we were there some time ago: https://www.noemamag.com/artificial-general-intelligence-is-...

We just keep moving the goalposts.

K0balt2• 2 years ago

I agree. The systems in place already solve generalized problems not directly represented in the training set or algorithm . That was, up until the last few years , the off the shelf definition of AGI.

And the systems in place do so at scales and breadths that no human could achieve.

That doesn’t change the fact that it’s effectively triple PHD uncle Jim, as in slightly unreliable and prone to bullshitting its way through questions, despite having a breathtaking depth and breadth of knowledge.

What we are making is not software in any normal sense of the word, but rather an engine to navigate the entire pool of human knowledge, including all of the stupidity, bias, and idiosyncrasies of humanity, all rolled up into a big sticky glob.

It’s an incredibly powerful tool, but it’s a fundamentally different class of tool. We cannot expect to apply conventional software processes and paradigms to LLM based tools any more than we could apply those paradigms to politics or child rearing and expect useful results.

netdevphoenix3• 2 years ago

The systems in place already solve generalized problems not directly represented in the training set or algorithm

Tell me a problem that an LLM can solve that is not directly represented in the training set or algorithm. I would argue that 99% of what commercial LLMs gets prompted about are stuff that already existed in the training set. And they still hallucinate half lies about those. When your training data is most the internet, it is hard to find problems that you haven't encountered before

HarHarVeryFunny12• 2 years ago

"Today’s most advanced AI models have many flaws, but decades from now, they will be recognized as the first true examples of artificial general intelligence."

Norvig seems to be using a loose technical definition of AGI, roughly "AI with some degree of generality", which is hard to argue with, although by that measure older GOFAI systems like SOAR might also qualify.

Certainly "deep learning" in general (connectionist vs symbolic, self-learnt representations) was a step in the right direction, and LLMs a second step, but it seems we're still a half dozen MAJOR steps away from anything similar to animal intelligence, with one critical step being moving beyond full dataset pre-training to new continuous learning algorithms.

m_ke5• 2 years ago

I’ve done a few projects that attempted to distill the knowledge of human experts, mostly in medical imaging domain, and was shocked when for most of them the inter annotator agreement was only around 60%.

These were professional radiologists with years of experience and still came to different conclusions for fairly common conditions that we were trying to detect.

So yes, LLMs will make mistakes, but humans do too, and if these models do so less often at a much lower cost it’s hard to not use them.

tomrod6• 2 years ago

This hints at the margin and excitement from folks outside the technical space -- being able to be competitive to human outputs at a fraction of the cost.

ethbr14• 2 years ago

That's the underappreciated truth of the computer revolution in practice.

At scale, computers didn't change the world because they did things that were already being computed, more quickly.

They changed the world because they decreased the cost of computing so much that it could be used for an entirely new class of problems. (That computing cost previously precluded its use on)

threeseed• 2 years ago

The problem is that how mistakes are made is crucial.

If it's a forced binary choice then sure LLMs can replace humans.

But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

m_ke5• 2 years ago

None of these were binary decisions, but classifying one of around 10-20 conditions or rating cases on a 1-5 scale.

In all cases the models trained on a lot of this feedback were more consistent and accurate than individual expert annotators.

IanCal6• 2 years ago

Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

Have you not seen an LLM say it doesn't know the answer to something? I just asked

"How do I enable a scroflpublaflex on a ggh connection?"

to O1 pro as it's what I had open.

Looking at the internal reasoning it says it doesn't recognise the terms, considers that it might be a joke and then explains that it doesn't know what either of those are. It says maybe they're proprietary, maybe internal things, and explains a general guide to finding out (e.g. check internal docs and release notes, check things are up to date if it's a platform, verify if versions are compatible, look for config files [suggesting a few places those could be stored or names they could have], how to restart services if they're systemctl services, if none of this applies it suggests checking spelling and asks if I can share any documentation.

This isn't unique or weird in my experience. Better models tend to be better at saying they don't know.

Shorel2• 2 years ago

But often there are many shades of grey e.g. a human may say I don't know and refer to someone else or do some research. Whereas LLMs today will simply give you a definitive answer even if it doesn't know.

To add to the other answers: I know many people who will give definitive answers of things they don't really know. They just rely on the fact you also don't know. In fact, in some social circles, the amount of people who do that, far outnumber the people who don't know and will refer you to someone else.

ksec• 2 years ago

Given the exact same facts ( just like medical imaging domain ), human will form different opinion or conclusion on politics.

I think what is not discussed enough is the assumption of assumption. [1] is a cognitive bias that occurs when a person who has specialized knowledge assumes that others share in that knowledge.

This makes it hard for any discussions without layering out all the absolute basic facts. Which has now more commonly known as First Principle in modern era.

In the four quadrants known and unknown. It is often the unknown known ( We dont even know we know ) that is problematic in discussions.

[1] Curse of knowledge - https://en.wikipedia.org/wiki/Curse_of_knowledge

ADeerAppeared3• 2 years ago

So yes, LLMs will make mistakes, but humans do too

Are you using LLMs though? Because pretty much all of these systems are fairly normal classifiers, what would've been called Machine Learning 2-3 years ago.

The "AI hype is real because medical AI is already in use" argument (and it's siblings) perform a rhetorical trick by using two definitions of AI. "AI (Generative AI) hype is real because medical AI (ML classifiers) is already in use" is a non-sequitur.

Image classifiers are very narrow intelligences, which makes them easy to understand and use as tools. We know exactly what their failure modes are and can put hard measurements on them. We can even dissect these models to learn why they are making certain classifications and either improve our understanding of medicine or improve the model.

...

Basically none of this applies to Generative AI. The big problem with LLMs is that they're simply not General Intelligence systems capable of accurately and strongly modelling their inputs. e.g. Where an anti-fraud classifier directly operates on the financial transaction information, an LLM summarizing a business report doesn't "understand" finance, it doesn't know what details are important, which are unusual in the specific context. It just stochastically throws away information.

m_ke5• 2 years ago

Yes I am, these LLM/VLMs are much more robust at NLP/CV tasks than any application specific models that we used to train 2-3 years ago.

I also wasted a lot of time building complex OCR pipelines that required dewarping / image normalization, detection, bounding box alignment, text recognition, layout analysis, etc and now open models like Qwen VL obliterate them with an end to end transformer model that can be defined in like 300 lines of pytorch code.

SoftTalker3• 2 years ago

This is why second opinions are widely used in any serious medical diagnosis.

Havoc3• 2 years ago

This take seems fundamentally wrong to me. As in opening premise.

We use humans for serious contexts & mission critical tasks all the time and they're decidedly fallible and their minds are basically black boxes too. Surgeons, pilots, programmers etc.

I get the desire for reproducible certainty and verification like classic programming and why a security researcher might push for that ideal, but it's not actually a requirement for real world use.

skydhash4• 2 years ago

Legal punishment is a great incentive to try to do your best job. You can reliably trust someone to act in one’s best interest.

protomolecule• 2 years ago

Maybe include in a prompt a threat of legal punishment? Sure somebody has already tried that and tabulated how much it improves scores on different benchmarks)

Havoc3• 2 years ago

LLM's response being best prediction of next token arguably isn't that far off from a human motivated to do their best. It's a fallible best effort either way.

And both are very far from the certainty the author seems to demand.

prisenco• 2 years ago

We've had 300,000 years to adapt to the specific ways in which humans are fallible, even if our minds are black boxes.

Humans fail in predictable and familiar ways.

Creating a new system that fails in unpredictable and unfamiliar ways and affording it the same control as a human being is dangerous. We can't adapt overnight and we may never adapt.

This isn't an argument against the utility of LLMs, but against the promise of "fire and forget" AI.

Havoc3• 2 years ago

Agreed that there shouldn't be automatic or even rapid reliance based on the parallels I drew to humans.

My point was more that falliability isn't the inherent show stopper the author makes it out to be.

snowwrestler2• 2 years ago

Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc.

What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess.

Even if we suppose that LLMs work similar to the human mind (a huge supposition!), LLMs still do not do their work like teams of humans. An LLM dreams and guesses, and it still falls to humans to check and verify.

Rigorous human work is actually a highly social activity. People interact using formal methods and that is what produces reliable results. Using an LLM as one of the social nodes is fine, but this article is about the typical use of software, which is to reliably encode those formal methods between humans. And LLMs don’t work that way.

Basically, we can’t have it both ways. If an LLM thinks like a human, then we should not think of it as a software tool like curl or grep or Linux or Apple Photos. Tools that we expect (and need) to work the exact same way every time.

725686• 2 years ago

"People do not just sit there thinking and then spitting out their best guess."

Well, if you are using AI like this, you are doing it wrong. Yes AI is imperfect, fallible, it sometimes hallucinates, but it is a freaking time saver (10x?). It is a tool. Don't expect a hammer to build you a cabinet.

TomK32• 2 years ago

Because human minds are fallible black boxes, we have developed a wide variety of tools that exist outside our minds, like spoken language, written language, law, standard operating procedures, math, scientific knowledge, etc.

Standard operating procedures are great but simplify it to checklists. Don't ever forget checklists which have proven vital for pilots and surgeons alike. And looking at the WHO Surgical Safety Checklist you might think "that's basic stuff" but apparently it is necessary and works https://www.who.int/teams/integrated-health-services/patient...

jvanderbot2• 2 years ago

This is a fantastic and thought-provoking response.

Thinking of humans as fallible systems and humanity and its progress as a self-correcting distributed computation / construction system is going to stick with me for a long time.

SoftTalker3• 2 years ago

What does it look like for fallible human minds to work on engineering an airplane? Things are calculated, recorded, checked, tested. People do not just sit there thinking and then spitting out their best guess.

People used to do this. The result was massively overbuilt structures, some of which are still with us hundreds of years later. The result was also underbuilt structures, which tended to collapse and maybe kill people. They are no longer around.

All of the science and math and process and standards in modern engineering is the solution humans came up with because our guesses aren't good enough. LLMs will need the same if they are to be relied upon.

chamomeal• 2 years ago

This is a really interesting perspective and a great point.

codingdave• 2 years ago

Human minds are far less black boxes than LLMs. There are entire fields of study and practice dedicated to understanding how they work, and to adjust how they work via medicine, drugs, education, therapy, and even surgery. There is, of course, a lot more to learn in all of those arenas, and our methods and practices are fallible. But acting as if it is the same level of black box is simply inaccurate.

bee_rider10• 2 years ago

They are much more of a black box than AI. There are whole fields around studying them—because they are hard to understand. We put a lot of effort into studying them… from the outside, because we had no other alternative. We were reduced to hitting brains with various chemicals and seeing what happened because they are such a pain to work with.

They are just a more familiar black box. AI’s are simpler in principle. And also entirely built by humans. Based on well-described mathematical theories. They aren’t particularly black-box, they are just less ergonomic than the human brain that we’ve been getting familiar with for hundreds of thousands of years through trial and error.

Closi2• 2 years ago

They are more of a black box - but humans are a black box that is perhaps more studied and that we have more experience in.

Although human behavior is still weird, and highly fallable! Despite best interventions (therapy, drugs, education), sometimes they still kill each other and we aren't 100% sure why, or how to solve it.

That doesn't mean that the same level of study can't be done on AI though, and they are much easier to adjust compared to the human brain (RLHF is more effective than therapy or drugs!).

nuancebydefault• 2 years ago

I would say human behavior is less predictable. That is one of the reasons why today it is rather easy to spot the bot responses, they tend to fit a certain predictable style, unlike the more unpredictable humans.

thuuuomas• 2 years ago

I tire of this disingenuous comparison. The failure modes of (experienced, professional) humans are vastly different than the failure modes of LLMs. How many coworkers do you have that frequently, wildly hallucinate while still performing effectively? Furthermore, (even experienced, professional) humans are known to be fallible & are treated as such. No matter how many gentle reminders the informed give the enraptured, LLMs will continue to be treated as oracles by a great many people, to the detriment of their application.

nullc• 2 years ago

Wildly hallucinating agents being treated as oracles is a human tradition.

bsenftner7• 2 years ago

If you expect the AI to do independent work, yes, it is a dead end.

These LLM AIs need to be treated and handled as what they are: idiot savants with vast and unreliable intelligence.

What does any advanced organization do when they hire a new PhD, let them loose in the company or pair them with experienced staff? When paired with experienced staff, they use the new person for their knowledge but do not let them change things on their own until much later, when confidence is established and the new staffer has been exposed to how things work "around here".

The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed. That means the path forward with this current state of LLM AIs is to pair them with people, personalized to their needs, and treat them as very smart idiot savants great for strategy and problem solving discussion, where the human users are driving the situation, using the LLM AIs like a smart assistant that requires validation - just like a real new hire.

There is an interactive state that can be achieved with these LLM AIs, like being in a conversation with experts, where they advise, they augment and amplify individual persons. A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today.

Alex39173• 2 years ago

The big difference with LLM AIs is they never graduate to an experienced staffer, they are always the idiot savant that is really dang smart but also clueless and needs to be observed.

Basically this. They already have vastly better-than-human ability at finding syntax errors within code, which on its own is quite useful; think of how many people have probably dropped out of CS as a major after staying up all night and failing to find a missing semicolon.

lionkor2• 2 years ago

I don't know of a single person who got so stuck on syntax errors that they quit

fire_lake• 2 years ago

Syntax checking is not an “AI” problem - use any compiler or linter.

rsynnott3• 2 years ago

… One odd thing I’ve noticed about the people who are very enthusiastic about the use of LLMs in programming is that they appear to be unaware of any _other_ programming tools. Like, this is a solved problem, more or less; code-aware editors have been a thing since the 90s (maybe before?)

dgfitz3• 2 years ago

I know a lot of people who dropped out of CS in college. Not a single one dropped out because of a semicolon syntax issue.

raincole• 2 years ago

think of how many people have probably dropped out of CS as a major after staying up all night and failing to find a missing semicolon.

... like a dozen? And in 100% cases it's their teacher's fault.

layer85• 2 years ago

They are still worse at finding syntax errors than the actual parser. And at best they could be equally good. So what’s the point?

lobsterthief• 2 years ago

I agree with all of what you said except this:

idiot savants with vast and unreliable intelligence.

Remember, intelligence !== knowledge. These LLMs indeed have vast and unreliable knowledge banks.

bsenftner7• 2 years ago

Yes, you are correct. They provide knowledge and the human is the operator of the intelligence portion.

wanderingstan• 2 years ago

In general conversation, “intelligence”, “knowledge”, “smartness”, “expertise”, etc are used mostly interchangeably.

If we want to get pedantic, I would point out that “knowledge” is formally defined as “justified true belief”, and I doubt we want to get into the quagmire of whether LLM’s actually have beliefs.

I took OP’s point in the casual meaning, i.e. that LLMs are like what I would call an “intelligent coworker”, or how one might call a Jeopardy game show contestant as intelligent.

skydhash4• 2 years ago

One of the core tenet of technology is that it makes the job less consuming of a person resources (time, strength,…). While I’ve read a lot of claims, I’ve yet to see someone make a proper argument on how LLMs can be such a tool.

A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today

More than the people who landed someone on the moon?

bsenftner7• 2 years ago

They would be capable of landing someone on the moon, if they chose to pursue that goal, and had the finances to do so. And they'd do so with fewer people too.

ethbr14• 2 years ago

Cited contextual information retrieval.

One of the obvious uses for current LLMs is as a smarter search tool against static knowledge collections.

Turns out, this is a real world problem in a lot of "fuzzy decision" scenarios. E.g. insurance claim adjudication

Status quo is to train a person over enough years that they can make these decisions reliably. (Because they've internalized all the documentation)

coliveira4• 2 years ago

It's even worse. AI is a really smart but inexperienced person who also lies frequently. Because AI is not accountable to anything, it'll always come up with a reasonable answer to any question, if it is correct or not.

belZaah• 2 years ago

To put it in other words: it is not clear when and how they hallucinate. With a person, their competence could be understood and also their limits. But a llm can happily give different answers based on trivial changes in the question with no warning.

Polizeiposaune• 2 years ago

Saying that they "lie" creates the impression that they have knowledge that they make false statements, and they intend to deceive.

They're not that capable. They're just bullshit artists.

LLM = LBM (large bullshit models).

oh_my_goodness8• 2 years ago

"AI is a really smart but inexperienced person who also lies frequently." Careful. Here "smart" means "amazing at pattern-matching and incredibly well-read, but has zero understanding of the material."

mattgreenrocks2• 2 years ago

It is a wonderful irony that AI makes competence all the more important.

It’s almost like all the thought leading that proclaimed the death of software eng was nothing but self-promotional noise. Huh, go figure.

scarface_745• 2 years ago

Hallucinations can be mostly eliminated with RAG and tools. I use NotebookLM all of the time to research through our internal artifacts, it includes citations/references from your documents.

Even with ChatGPT you can ask it to find web citations and if it uses the Python runtime to find answers, you can look at the code.

And to prevent the typical responses - my company uses GSuite so Google already has our IP, NotebookLM is specifically approved by my company and no Google doesn’t train on your documents

_heimdall2• 2 years ago

That sounds mostly like an incentives problem. If OpenAI, Anthropic, etc decide their LLMs need to be accurate they will find some way of better catching hallucinations. It probably will end up (already is?) being yet another LLM acting as a control structure trying to fact check responses before they are sent to users though, so who knows if it will work well.

Right noe there's no incentive though. People keep paying good money to use these tools despite their hallucinations, aka lies/gas lighting/fake information. As long as users don't stop paying and LLM companies don't have business pressure to lean on accuracy as a market differentiator, no one is going to bother fixing it.

uludag• 2 years ago

It's actually even worse than that: the current trend of AI is transformer-based deep learning models that use self-attention mechanisms to generate token probabilities, predicting sequences based on training data.

If only it was something which we could ontologically map onto existing categories like servants or liars...

kraftman3• 2 years ago

If I had a senior member of the team that was incredibly knowledgeable but occasionally lied, but in a predictable way, I would still find that valuable. Talking to people is a very quick and easy way to get information about a specific subject in a specific context, so I could ask them targetted questions that are easy to verify, the worst thing that happens is I 'waste' a conversation with them.

layer85• 2 years ago

Lying means stating things as facts despite knowing or believing that they are false. I don’t think this accurately characterizes LLMs. It’s more like a fever dream where you might fabulate stuff that appears plausibly factual in your dream world.

api4• 2 years ago

After using them for a long time I am convinced they have no true intelligence beyond what is latent in training data. In other words I think we are kind of fooling ourselves.

That being said they are very useful. I mostly use them as a far superior alternative to web search and as a kind of junior research assistant. Anything they find must be checked of course.

I think we have invented the sci-fi trope of the AI librarian of the galactic archive. It can’t solve problems but it can rifle through the totality of human knowledge and rapidly find things.

It’s a search engine.

weakfish2• 2 years ago

I mean, it’s known that there’s no intelligence if you simply look at how it works on a technical level - it’s a prediction of the next token. That wasn’t really ever in question as to whether they have “intelligence”

returnInfinity• 2 years ago

But Ilya has convinced himself and many others that predicting the next token is intelligence.

zcw100• 2 years ago

And a plagiarism machine. It's like a high school student that thinks they can change a couple of words, make sure it's grammatically correct and it's not plagiarism because it's not an exact quote....either that or it just completely makes it up. I think LLMs will be revolutionary but just not in the way people think. It may be similar to the Gutenberg press. Before the printing press words were precious and closely held resources. The Gutenberg press made words cheap and abundant. Not everyone thought it was a good thing at the time but it changed everything.

coffeefirst• 2 years ago

The problem is it's still a computer. And that's okay.

I can ask the computer "hey I know this thing exists in your training data, tell me what it is and cite your sources." This is awesome. Seriously.

But what that means is you can ask it for sample code, or to answer a legal question, but fundamentally you're getting a search engine reading something back to you. It is not a programmer and it is not a lawyer.

The hype train really wants to exaggerate this to "we're going to steal all the jobs" because that makes the stock price go up.

They would be far less excited about that if they read a little history.

insane_dreamer4• 2 years ago

"we're going to steal all the jobs"

It won't steal them all, but it will have a major impact by stealing the lower level jobs which are more routine in nature -- but the problem is that those lower level jobs are necessary to gain the experience needed to get to the higher level jobs.

It also won't eliminate jobs completely, but it will greatly reduce the number of people needed for a particular job. So the impact that it will have on certain trades -- translators, paralegals, journalists, etc. -- is significant.

ethbr14• 2 years ago

The thing that makes the smarter search use case interesting is how LLMs are doing their search result calculations: dynamically and at metadata scales previously impossible.

LLM-as-search is essentially the hand-tuned expert systems AI vs deep learning AI battle all over again.

Between natural language understanding and multiple correlations, it's going to scale a lot further than previous search approaches.

cshores• 2 years ago

I find it fascinating that I can achieve about 85-90% of what I need for simple coding projects in my homelab using AI. These projects often involve tasks like scraping data from the web and automating form submissions.

My workflow typically starts with asking ChatGPT to analyze a webpage where I need to authenticate. I guide it to identify the username and password fields, and it accurately detects the credential inputs. I then inform it about the presence of a session cookie that maintains login persistence. Next, I show it an example page with links—often paginated with numbered navigation at the bottom—and ask it to recognize the pattern for traversing pages. It does so effectively.

I further highlight the layout pattern of the content, such as magnet links or other relevant data presented by the CMS. From there, I instruct it to generate a Python script that spiders through each page sequentially, navigates to every item on those pages, and pushes magnet links directly into Transmission. I can also specify filters, such as only targeting items with specific media content, by providing a sample page for the AI to analyze before generating the script.

This process demonstrates how effortlessly AI enables coding without requiring prior knowledge of libraries like beautifulsoup4 or transmission_rpc. It not only builds the algorithm but also allows for rapid iteration. Through this exercise, I assume the role of a manager, focusing solely on explaining my requirements to the AI and conducting a code review.

insane_dreamer4• 2 years ago

with vast and unreliable intelligence

I would say "knowledge" rather than "intelligence"

The key feature of LLMs is the vast amounts of information and data they have access to, and their ability to quickly process and summarize, using well-written prose, that information based on pattern matching.

ethbr14• 2 years ago

This is what LLM (and AI in general) naysayers are missing.

LLMs will likely never get us to 100% solutions on a large class of problems.

But! A lot of problems can be converted into versions with a subcomponent that LLMs can solve 100% of.

And the fusion of LLMs doing 100% of that subportion + humans doing the remainder = increased productivity.

Re-engineering problems to be LLM-tolerant, then using LLMs to automate that portion of the problem, is the winning approach.

cess112• 2 years ago

So you think machines running statistical inference have awareness.

That's quite the embarassment if you actually mean it.

cactacea2• 2 years ago

A group of individuals adept with use of such an idiot savant enhanced environment would be incredibly capable. They'd be a force unseen in human civilization before today.

I'm sorry but your comment is a good example of the logical shell game many people play with AI when applying it to general problem solving. Your LLM AI is both an idiot and an expert somehow? Where is this expertise derived from and why should you trust it? If LLMs were truly as revolutionary as all the grifters would have you believe then why do we not see "forces unseed in human civilization before today" by humans that employ armies of interns? That these supposed ubermensch do not presently exist is firm evidence in support of current AI being a dead end in my opinion.

Humans are infinitely more capable than current AI, the limiting factor is time and money. Not capability!

dylan6043• 2 years ago

Your LLM AI is both an idiot and an expert somehow?

Maybe you are unfamiliar with the term idiot savant?

monkeynotes2• 2 years ago

I was so stupid when GPT3 came out. I knew so little about token prediction, I argued with folks on here that it was capable of so many things that I now understand just aren't compatible with the tech.

Over the past couple of years of educating myself a bit, whilst I am no expert I have been anticipating a dead end. You can throw as much training at these things as you like, but all you'll get is more of the same with diminishing returns. Indeed in some research the quality of responses gets worse as you train it with more data.

I am yet to see anything transformative out of LLMs other than demos which have prompt engineers working night and day to do something impressive with. Those Sora videos took forever to put together, and cost huge amounts of compute. No one is going to make a whole production quality movie with an LLM and disrupt Hollywood.

I agree, an LLM is like an idiot savant, and whilst it's fantastic for everyone to have access to a savant, it doesn't change the world like the internet, or internal combustion engine did.

OpenAI is heading toward some difficult decisions, they either admit their consumer business model is dead and go into competing with Amazon for API business (good luck), become a research lab (give up on being a billion dollar company), or get acquired and move on.

pnut• 2 years ago

Criticisms like this are levied against an excessively narrow (obsolete?) characterisation of what is happening in the AI space currently.

After reading about o3's performance on ARC-AGI, I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year. I'm genuinely surprised at how myopic HN commentary is on this topic in general. Maybe because the implications are almost unthinkably profound.

Anyway, OpenAI, Anthropic, Meta, and everyone else are well aware of these types of criticisms, and are making significant, measurable progress towards architecturally solving the deficiencies.

https://arcprize.org/blog/oai-o3-pub-breakthrough

jokethrowaway3• 2 years ago

Nah, the trick with o3 solving IQ tests seems to be that they bruteforce solutions and then pick the best option. That's why calls that are trivial for humans end up costing a lot.

It still can't think and it won't think.

LANGUAGE models (keyword: language) is a language model, it should be paired with a reasoning engine to translate the inner thought of the machine into human language. It should not be the source of decisions because it sucks at doing so, even though the network can exhibit some intelligence.

We will never have AGI with just a language model. That said, most jobs people do are still at risk, even with chatgpt-3.5 (especially outside of knowledge work, where difficult decisions need to be taken). So we'll see the problems with AGI and the job market way earlier than AGI, as soon as we apply robotics and vision models + chatgpt 3.5 level intelligence. Goodbye baristas, goodbye people working in factories.

Let's start working on a reasoning engine so we can replace those pesky knowledge workers too.

rafaelmn• 2 years ago

Reading the o1 announcement you could have been saying the same thing a year ago yet it's worse than Claude in practice and if it was all that's available - I wouldn't even use it if it was free - it's that bad.

If OpenAI has demonstrated one thing is that they are a hype production machine and they are probably getting ready for next round of investment. I wouldn't be surprised if this model was equally useless as o1 when you factor in performance and price.

At this point they are completely untrustworthy and untill something lands publicly for me to test it's safe to ignore their PR as complete BS.

portaouflop3• 2 years ago

Any day now!

mbesto3• 2 years ago

I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year.

People are flippantly dismissive of the inherent limits because there ARE inherent limitations of the technology.

Maybe because the implications are almost unthinkably profound.

Maybe because the stuff you're pointing to are just benchmarks and the definitions around things like AGI are flawed (and the goalposts are constantly moving, just like the definition of autonomous driving). I use LLMs roughly 20-30x a day - they're an absolutely wonderful tool and work like magic, but they are flawed for some very fundamental reasons.

netdevphoenix3• 2 years ago

You remember when Google was scared to release LLMs? You remember that Googler that got fired because he thought the LLM was sentient?

There is likely a couple of surprised still left in LLMs but no one should think that any present technology in its current state or architecture will get us to AGI or anything that remotely resembles it.

gosub100• 2 years ago

Maybe because the implications are almost unthinkably profound.

laundering stolen IP from actual human artists and researchers, extinguishing jobs, deflecting responsibility for disasters. yeah, I can't wait for these "profound implications" to come to fruition!

formerly_proven• 2 years ago

It doesn’t really matter. “It works and is cost/resource-effective at being an AGI” is a fundamentally uninteresting proposition because we’re done at that point. It’s like debating how we’re going to deal with the demise of our star; we won’t, because we can’t.

fabianhjr• 2 years ago

The question of whether a computer can think is no more interesting than the question of whether a submarine can swim. ~ Edsger W. Dijkstra

LLMs / Generative Models can have a profound societal and economic impact without being intelligent. The obsession with intelligence only make their use haphazard and dangerous.

It is a good thing court of laws have established precedent that organizations deploying LLM chatbots are responsible for their output (Eg, Air Canada LLM chatbot promising a non-existent discount being responsibility of Air Canada)

Also most automation has been happening without LLMs/Generative Models. Things like better vision systems have had an enormous impact with industrial automation and QA.

n144q3• 2 years ago

I'll believe that when ChatGPT stops making up APIs that have never ever existed in the history of a library.

The dumbest intern doesn't do that.

Which is the entire point of the article that your comment fail to address.

zwnow• 2 years ago

Lmao, you are the type of person actually believing these silicon valley bs. o3 is far, far away from AGI.

aaroninsf• 2 years ago

Indeed, this put me immediately in mind of Ximm's Law:

Every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.

Lemma: any statement about AI which uses the word "never" to preclude some feature from future realization is false.

cootsnuck2• 2 years ago

But o3 is just a slightly less stupid idiot savant...it still has to brute force solutions. Don't get me wrong, it's cool to see how far that technique can get you on a specific benchmark.

But the point still stands that these systems can't be treated as deterministic (i.e. reliable or trustworthy) for the purposes of carrying out tasks that you can't allow "brute forced attempts" for (e.g. anything where the desired outcome is a positive subjective experience for a human).

A new architecture is going to be needed that actually does something closer to our inherently heuristic based learning and reasoning. We'll still have the stochastic problem but we'll be moving further away from the idiot savant problem.

All of this being said, I think there's plenty of usefulness with current LLMs. We're just expecting the wrong things from them and therefore creating suboptimal solutions. (Not everyone is, but the most common solutions are, IMO.)

The best solutions need to be rethinking how we typically use software since software has been hinged upon being able to expect (and therefore test) dertiministic outputs from a limited set of user inputs.

I work for an AI company that's been around for a minute (make our own models and everything). I think we're both in an AI hype bubble while simultaneously underestimating the benefits of current AI capabilities. I think the most interesting and potentially useful solutions are inherently going to be so domain specific that we're all still too new at realizing we need to reimagine how to build with this new tech in mind. It reminds me of the beginning of mobile apps. It took awhile for most us to "get it".

JohnMakin• 2 years ago

After reading about o3's performance on ARC-AGI, I strongly suspect people will not be so flippantly dismissive of the inherent limits of these technologies by this time next year.

If I wasn't so slammed with work I have half a mind to go dredge up at least a dozen posts that said the same thing last year, and the year before. Even OpenAI has been moving the goalposts here.

tsurba• 2 years ago

My favorite quote in this topic:

”If intelligence lies in the process of acquiring new skills, there is no task X that solving X proves intelligence”

IMO it especially applies to things like solving a new IQ puzzle, especially when the model is pretrained for that particular task type, like was done with ARC-AGI.

For sure, it’s very good research to figure out what kind of tasks are easy for humans and difficult for ML, and then solve them. The jump in accuracy was surprising. But still in practice the models are unbeliavably stupid and lacking in common sense.

My personal (moving) goalpost for ”AGI” is now set to whether a robot can keep my house clean automatically. Its not general intelligence if it can’t do the dishes. And before physical robots, being less of a turd at making working code would be a nice start. I’m not yet convinced general purpose LLMs will lead to cost-effective solutions to either vs humans. A specifically built dish washer however…

cormackcorn• 2 years ago

Myopic? You must be under 20 years old. For those of us who have been in tech for over four decades the OPs assessment is exactly the right framing.

benterix2• 2 years ago

After reading about o3's performance

I heard that people still believing in OpenAI hype exist but I haven't met any IRL.

FuriouslyAdrift• 2 years ago

LLMs are fuzzy compression with a really good natural language parser...

danielbln3• 2 years ago

And strong in-context learning, the true killer feature.

ALittleLight• 2 years ago

You really shouldn't say LLMs "never graduate" to experienced staff - rather that they haven't yet. But there are recent and continuing improvements in the ability of the LLMs, and in time, perhaps a small amount of time, this situation may flip.

bsenftner7• 2 years ago

I'm talking about the current SOTA. In the future, all bets are off. For today, they are very capable when paired with a capable person, and that is how one uses them successfully today. Tomorrow will be different, of course.

brookst3• 2 years ago

I think you’ve exactly captured the two disparate views we see on HN:

1. LLMs have little value, are totally unreliable, will never amount to much because they don’t learn and grow and mature like people do., so they cannot replace a person like me who is well advanced in a career.

2. LLMs are incredible useful and will change the world because they excel at entry level work and can replace swaths of relatively undifferentiated information workers. LLM flaws are not that different from those workers’ flaws.

I’m in camp 2, but I appreciate and agree with the articulation of why they will not replace every information worker.

layer85• 2 years ago

This all sounds plausible, but personally I find being paired to a new idiot-savant hire who never learns anything from the interaction incredibly exhausting. It can augment and amplify one’s own capabilities, but it’s also continuously frustrating and cumbersome.

iambateman• 2 years ago

While these folks waste breath debating whether AI is useful, I’m going to be over here…using it.

I use AI 100 times a day as a coder and 10,000 times a day in scripts. It’s enabled two specific applications I’ve built which wouldn’t be possible at single-person scale.

There’s something about the psychology of some subset of the population that insists something isn’t working when it isn’t _quite_ working. They did this with Wikipedia. It was evident that Wikipedia was 99% great for years before this social contingent was ready to accept it.

Mistletoe• 2 years ago

But please accept that you are in a small subset of people that it is very useful to. Every time I hear someone championing AI, it is a coder. AI is basically useless to me, it is just a convoluted expensive google search.

tossandthrow4• 2 years ago

I use ai to care for my plants, to give me recipes for pan pancakes, to help me fix my coffee machine.

LLMs as a popularized thing is just about 2 years old. It is still mainly early adopters.

For smartphones it might have taken 10 to 15 years to gain widespread traction.

I think it is safe to say that we are only scratching the surface.

davidmurdoch• 2 years ago

Do you not use it to try learning new things? I use it to help get familiar with new software (recently for FreeCAD), or new concepts (passive speaker crossover design).

josh2600• 2 years ago

Wow, this is a wild opinion. I wonder how many people you’ve talked to about this?

I know tons of people in my social groups that love AI and use it every day in it’s current form.

herval6• 2 years ago

it's _extremely_ useful for lawyers, arguably even more so than for coders, given how much faster they can do stuff. They're also extremely useful for anyone who writes text and wants a reviewer. Also capable to execute most daily activities of some roles, such as TPMs.

It's still useful to a small subset of all those professions - the early adopters. Same way computers were useful to many professionals before the UI (but only a small fraction of them had the skillset to use terminals)

y1n02• 2 years ago

My wife uses it almost as much as me which isn’t quite daily. She is not a coder whatsoever.

I’ll ask her what her use cases are and reply here later if I don’t forget.

umanwizard• 2 years ago

Walk into any random coffee shop in America where people are working on their laptops and you will see some subset of them on ChatGPT. It’s definitely not just coders who are finding it useful.

Ukv2• 2 years ago

Particularly given the article's target is "systems based on large neural networks" and not specifically LLMs, I'd claim there are a vast number of uncontroversially beneficial applications: language translation, video transcription, material/product defect detection, weather forecasting/early warning systems, OCR, spam filtering, protein folding, tumor segmentation, drug discovery and interaction prediction, etc.

stickfigure• 2 years ago

convoluted expensive google search

I'd call it a working google search, unlike, you know, google these days.

Actually google's LLM-based search results have been getting better, so maybe this isn't the end of the line for them. But for sophisticated questions (on noncoding topics!) I still always go to chatgpt or claude.

FrustratedMonky7• 2 years ago

It's being used in drive through windows. In movies, in graphic design, pod casts, music, etc... 'entertainment' industry.

And HN, it isn't just a few odd balls on HN championing it. I wish there was way to get a sentiment analysis of HN, it seems there are lot more people using it than not using it.

And, what about the silent majority, the programmers that don't hang out on HN? I hear colleagues talk about it all the time.

The impact is here, whether they are self directed or not, or whether there are still a few people not using it.

sebastiansm• 2 years ago

Yesterday ChatGPT helped me to elaborate a skincare routine for my wife with multiple serums and creams that she received for Christmas. She and I had no idea when to apply, how to combine or when to avoid combination of some of those products.

I could have google it myself in the evenings and had the answer in a few days of research, but with o1 in a 15min session my wife had had a solid weekly routine, the reasoning about those choices and academic papers with research about those products. (Obviously she knows a lot about skincare in general, so she had the capacity to recognize any wrong recommendation).

Nothing game changer, but is great to save lots of time in this kind of tasks.

drooby• 2 years ago

convoluted expensive google search

Interesting, I'm the opposite now. Why would I click a couple links to read a couple (verbose) blog posts when I can read a succinct LLM response. If I have low confidence in the quality of the response then I supplement with Google search.

I feel near certain that I am saving time with this method. And the output is much more tuned to the context and framing of my question.

Hah, take for example my last query in ChatGPT:

Are there any ancient technologies that when discovered furthered modern understanding of its field?

ChatGPT gave some great responses, super fast. Google also provides some great results (though some miss the mark), but I would need to parse at least three different articles and condense the results.

To be fair, ChatGPT gives some bad responses too.. But both an LLM and Google search should be used in conjunction to perform a search at the same time.

Use LLMs as breadth-first search, and Google as depth-first search.

paulcole• 2 years ago

Every time I hear someone championing AI, it is a coder

The argument I make is why aren’t more people finding ways to code with AI?

I work in a leadership role at a marketing agency and am a passable coder for scripts using Python and/or Google Apps Scripts. In the past year, I’ve built more useful and valuable tools with the help of AI than I had in the 3 or so years before.

We’re automating more boring stuff than ever before. It boggles my mind that everybody isn’t doing this.

In the past, I was limited by technical ability because my knowledge of our business and processes was very high. Now I’m finding that technical ability isn’t my limitation, it’s how well I can explain our processes to AI.

mbesto3• 2 years ago

Not a coder here (although I can code). I use LLMs 15+ time a day.

tiahura4• 2 years ago

I’m a lawyer and AI has become deeply integrated into my work.

AnotherGoodName• 2 years ago

I'd argue that's just because coders are first to line up for this.

There was a different thread on this site i read where a journalist used the wrong units of measurement (kilowatts instead of killowatt-hours for energy storage). You could paste the entire article into chatGPT with a prompt "spot mistakes in the following; [text]" and get an appropriate correction for this and similar mistakes the author made.

As in there's journalists right now posting articles with clear mistakes that could have been proof read more accurately than they were if they were willing to use AI. The only excuse i could think of is resistance to change. A lot of professions right now could do their job better if they leant on the current generation of AI.

oytis• 2 years ago

In my bubble coders find LLMs least useful. After all we already have all kinds of fancy autocomplete that works deterministically and doesn't hallucinate - and still not everyone uses it.

When I use LLMs, I use it exactly as Google search on steroids. It's great for providing a summary on some unknown topic. It doesn't matter if it gets it wrong - the main value is in keywords and project names, and one can use the real Google search from there.

And it isn't expensive if you are using the free version

giraffe_lady4• 2 years ago

These are different social contingents I think. At least for me I was super on board with wikipedia because as you say the use to me was immediate and certain. AI I have tried every few months for the last two years but I still haven't found a strong use for it. It has changed nothing for me personally except making some products I use worse.

llm_trw2• 2 years ago

Have you paid for it?

Loughla4• 2 years ago

AI is only useful for me if I have a good idea of what the answer might already be, or at least what it absolutely can't be.

It helps me get to an answer a little bit quicker, but it doesn't perform any absolutely groundbreaking work for me.

xvector4• 2 years ago

The Wikipedia analogy strikes true.

Generally people are resistant to change and the average person will typically insist new technologies are pointless.

Electricity and the airplane were supposed to be useless and dangerous dead ends according to the common person: https://pessimistsarchive.org/

But we all like to think we have super unique opinions and personalities, so "this time it's different."

When the change finally happens, people go about their lives as if they were right all along and the new technology is simply a mysterious and immutable fixture of reality that was always there.

toddmorey• 2 years ago

I don’t think that was the common person, nor do I think the common person today thinks AI will be useless.

wat100003• 2 years ago

People also thought the Segway was a useless dead end and they were right.

bpfrh• 2 years ago

There is a vast difference between arguments like "Phones have been accused of ruining romantic interaction and addicting us to mindless chatter" and "current AI has problems generating accurate information and can't replace researching things by hand for complicated or niche topics and there is reason to believe that the current architecture may not solve this problem"

That aside optimist are also not always right, otherwise we would have cold fusion already and have a base on mars.

rsynnott3• 2 years ago

But we all like to think we have super unique opinions and personalities, so "this time it's different."

Are you suggesting that anything which is hyped is the future? Like, for every ten heavily-hyped things, _maybe_ one has some sort of post-hype existence.

coliveira4• 2 years ago

The pessimist is not wrong. In fact he's right more frequently than wrong. Just look at a long list of inventions. How many of them were so successful as the car or the airplane? Most of them were just passing fads that people don't even remember anymore. So if you're asking who is smarter, I would say the pessimist is closer to the truth, but the optimist who believed in something that really became successful is now remembered by everyone.

jdbernard• 2 years ago

This argument is very prone to survivorship bias. Of course, when we think back to the hyped technologies of the past we are going to remember mostly those that justified the hype. The failures get forgotten. The memory of social discourse fades extremely quickly, much faster than, for example, pop culture or entertainment.

i_love_retros2• 2 years ago

It's enabled two specific applications I’ve built which wouldn’t be possible at single-person scale.

I'd love to hear more about how you utilised AI for this.

Personally I'm struggling to find it more useful than a slightly fancy code completion tool

broast• 2 years ago

slightly fancy code completion tool

Does this alone not increase your productivity exponentially? It does mine. I personally read code faster than I write it so it is an undeniable boon.

wbazant• 2 years ago

You could try aider, or another tool/workflow where you provide whole files and ask for how they should be changed - very different from code completion type tools!

hbn• 2 years ago

Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.

I have no interest in learning the horrible unintuitive UX of every CLI I interact with, I'd much rather just describe in English what I want and have the computer figure it out for me. It has practically never failed me, and if it does I'll know right away and I can fall back to the old method of doing it manually. For now it's saving me so much time with menial, time-wasting day-to-day tasks.

jghn2• 2 years ago

I had a debate recently with a colleague who is very skeptical of LLMs for every day work. Why not lean in on searching Google and cross referencing answers, like we've done for ages? And that's fine.

But my counterargument is that what I find to be so powerful about the LLMs is the ability to refine my question, narrow in on a tangent and then pull back out, etc. And *then* I can take its final outcome and cross reference it. With the old way of doing things, I often felt like I was stumbling in the dark trying to find the right search string. Instead I can use the LLM to do the heavy lifting for me in that regard.

ADeerAppeared3• 2 years ago

Anyone who says AI is useless

Most of those people are a bit bad at making their case. What they mean but don't convey well is that AI is useless for it's proclaimed uses.

You are correct that LLMs are pretty good at guessing this kind of well-documented & easily verifiable but hard to find information. That is a valid use. (Though, woe betide the fool who uses LLMs for irreversible destructive actions.)

The thing is though, this isn't enough. There just aren't that many questions out there that match those criteria. Generative AI is too expensive to serve that small a task. Charging a buck a question won't earn the $100 billion OpenAI needs to balance the books.

Your use case gets dismissed because on it's own, it doesn't sustain AI.

hyhconito• 2 years ago

You're still doing it the hard way. I just use Handbrake.

Pick a hammer, not a shitty hammer factory to assemble bits of hammer.

arkh2• 2 years ago

if it does I'll know right away and I can fall back to the old method of doing it manually

It's well and ok with things you can botch with no consequence other than some time wasted. But I've bricked enough VMs trying commands I did not understand to know that if you need to not fuck up something you'll have to read those docs and understand them. And hope they're not out of date / wrong.

dangoodmanUT• 2 years ago

Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.

The best ffmpeg and regex command generators

jv981• 2 years ago

try asking an LLM how to add compressed data to a tar through standard input, see how that goes (don't forget to check the answer :)

cootsnuck2• 2 years ago

I use LLMs to help me with ffmpeg commands more than I care to admit

momentoftop3• 2 years ago

Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.

It's useful for that yes, but I'd rather just live in a world where we didn't have such disasters of CLI that are git and ffmpeg.

LLMs are very useful for generating the obscure boilerplate needed because the underlying design is horrible. Relying on it means acquiescing to those terrible designs rather than figuring out redesigns that don't need the LLMs. For comparison, IntelliJ is very good at automating all the boilerplate generation that Java imposes on me, but I'd rather we didn't have boilerplate languages like Java, and I'd rather that IntelliJ's boilerplate generation didn't exist.

I fear in many cases that if an LLM is solving your problem, you are solving the wrong problem.

skydhash4• 2 years ago

Anyone who says AI is useless never had to do the old method of cobbling together git and ffmpeg commands from StackOverflow answers.

These days, I'm more likely to read the manual pages and take notes on interesting bits. If I'm going to rely on some tooling for some time, dedicating a few hours of reading is a good trade-off for me. No need to even remember everything, just the general way it solves the problem. Anything more precise is captured in notes, scripts, shell history,... I dare anyone to comes out with an essay like this from LLMs: https://karthinks.com/software/avy-can-do-anything/

wruza• 2 years ago

We can't test/review these apps though, can we?

I'm asking not for snark, but because when AI gives me something not _quite_ working, it requires much more time than what a "every 6 minutes in 10 hour work day" frame would allow to investigate. I just wonder if maybe you're pasting it as is and don't care about correctness if the happy path sort of works. Speaking of subsets, coders who did that before AI were also quite a group.

There must be something that explains the difference in our experiences. Apologies for the fact that my only idea is kinda negative. I understand the potential hyperbola here, but it doesn't explain much. I can stand AI BS once a day, maybe twice, before uncontrollably cursing into the chat.

thomashop2• 2 years ago

Why not write tests with AI, too? Since using LLMs as coding assistants, my codebases have much more thorough documentation, testing and code coverage.

Don't start when you're already in a buggy dead-end. Test-driven development with LLMs should be done right from the start.

Also keep the code modular so it is easy to include the correct context. Fine-grained git commits. Feature-branches.

All the tools that help teams of humans of varying levels of expertise work together.

mistercheph2• 2 years ago

You may have enough expertise in your field that when you have a question, you know where to start looking. Juniors and students encounter dozens of problems and question per hour that fall into the unknown unknown category

croes7• 2 years ago

Are you still a coder when you use AI 100 times a day?

AI is a type of outsourcing, you became a customer.

mbernstein2• 2 years ago

Not outsourcing at all - you're are an engineer using the tools that make sense to solve a problem. The core issue with identifying as just a coder is that code is just one of many potential tools to solve a problem.

nlh• 2 years ago

If you're sitting in front of the keyboard, inputting instructions and running the resulting programs, yes you are still a coder. You're just move another layer up on the stack.

The same type of argument has been made for decades -- when coders wrote in ASM, folks would ask "are you still a coder when you use that fancy C to make all that low-level ASM obsolete?". Etc etc.

owenpalmer3• 2 years ago

Are you still a coder when you use libraries or frameworks? You didn't write the code yourself, you're just outsourcing it.

stronglikedan• 2 years ago

Have you tried a few? If so, which do you prefer? If not, which do you use? I'm a little late to the party, and the current amount of choices is quite intimidating.

airstrike• 2 years ago

I imagine you're asking about coding help. For that, I think you should qualify any answer you get with the user's most commonly used language (and framework, if applicable).

In my experience, Claude Sonnet 3.5 (3.6?) has been unbeatable. I use it for Rust. Making sense of compiler errors, rubberducking, finding more efficient ways to write some function and, truth be told, some times just plain old debugging. More than once, I've been able to dump a massive module onto the chat context and say "look, I'm experiencing this weird behavior but it's really hard to pin down what's causing it in this code" and it pointed to the exact issue in a second. That alone is worth the price of admission.

Way better than ChatGPT 4o and o-1, in my experience, despite me saying the exact opposite a few months ago.

esafak4• 2 years ago

Try Cody. It integrates with your IDE, understands your code base, and lets you pick the LLM.

monkeynotes2• 2 years ago

This isn't about if LLMs are useful, it's about how useful can they become. We are trying to understand if there is a path forward to transformative tech, or are we just limited to a very useful tool.

It's a valid conversation after ~3 years of anticipating the world to be disrupted by this tech. So far it has not delivered.

Wikipedia did not change the world either, it's just a great tool that I use all the time

As for software, it performs ok. I give up on it most of the time if I am trying to write a whole application. You have to acquire a new skill, prompt engineering, and feverish iteration. It's a frustrating game of whack-a-mole and I find it quicker to write the code myself and just have the LLM help me with architecture ideas, bug bashing, and it's also quite good at writing tests.

I'd rather know the code intimately so I can more quickly debug it than have an LLM write it and just trust it did it well.

wenc4• 2 years ago

By the way, Wikipedia did change the world. Some of the most important inventions are the ones we don’t notice.

nemo44x• 2 years ago

Peter Thiel talked about this years ago in his book From 0 to One. His key insight, which we're seeing today, is that AI tools will work side-by-side with people and enhance their productivity to levels never imagined. From helping with some basic tasks ("write an Excel script that transforms this table from this format to this new format") to helping write programs, it's a tool that aids humans in getting more things done than previously possible.

wodderam2• 2 years ago

To me this is besides the point.

LLMs obviously have use cases but the market has practically priced in "AGI".

The danger is not that LLMs take jobs. The danger is that we are in a massive bubble and while these are nice tools they are not worth anything close to the trillions of dollars bet on them.

IMO the psychology at work here is basically denial that we can both be in the biggest bubble of all time in terms of dollars and LLMs are useful. Just not THAT useful.

dabauws• 2 years ago

Why should we care about whether we're in a market bubble or not, especially if one does not have their own money staked in the bubble? It's somebody else's capital that's at risk, no? If they're wrong, let them reap the consequences.

(cue rebuttal based on systemic consequences / financial bailouts etc, but you know what I mean; also, the dotcom bubble deflation didn't require a bailout)

paxys• 2 years ago

Every piece of technology is a "dead end" until something better replaces it. That doesn't mean it can't be useful or revolutionary.

tu7001• 2 years ago

We all know what's an answer when there is a question mark in the title.

IanHalbwachs• 2 years ago

https://en.wikipedia.org/wiki/Betteridge%27s_law_of_headline...

btbuildem• 2 years ago

Interesting read -- and a correct take, given the software development perspective. In that context, LLM-based AI is faulty, unpredictable, and unmanageable, and not ready for mission-critical applications.

If you want to argue otherwise, do a quick thought experiment first: would you let an LLM manage your financial affairs (entirely, unsupervised)? Would you let it perform your job while you receive the rewards and consequences? Would you be comfortable to give it full control of your smart home?

There are different sets of expectations put on human actors vs autonomous systems. We expect people to be fallible and wrong some of the time, even if the individuals in question can't/won't admit it. With a software-based system, the expectations are that it will be robust, tested, and performing correctly 100% of the time, and when a fault occurs, it will be clear, marked with yellow tape and flashing lights.

LLM-based AIs are sort of insidious in that they straddle this expectation gap: the emergent behaviour is erratic, projecting confident omniscience, while often hallucinating and plain wrong. However vague, the catch-all term "AI" still implies "computer system" and by extension "engineered and tested".

cesaref2• 2 years ago

It's a bad example. Lots of finance firms use AI to manage their financial affairs - go and investigate what is currently considered state of the art for trading algorithms.

Now if you substituted something safety critical instead, say, running a nuclear power station, or my favourite currently in use example, self driving cars, then yes, you should be scared.

galleywest200• 2 years ago

go and investigate what is currently considered state of the art for trading algorithms.

These are not LLMs but algorithms written and designed by human minds. It is unfortunate that AI has become a catch-all word for any kind of machine learning.

wavemode3• 2 years ago

He said LLM, not just any AI

ryukoposting• 2 years ago

Makes me wonder how they detect market manipulation and fraud. Trivial activities, like marking the close, probably aren't hard to detect, but I imagine that some kind of ML thingy is involved in flagging accounts for manual inspection.

msabalau• 2 years ago

Pragmatically, "AI" will mean (and for, many people already does mean) stochastic and fallible.

If your users are likely to be AI illiterate and mistakenly feel that an AI app is reliable and suitable for mission critical applications when it isn't, that is a risk you mitigate.

But it seems deeply unserious of the author to just assert that mission-critical software is the only "serious context" and the only thing matters, and therefore AI is dead end. "Serious, mission critical" apps are just going to be a niche in the future.

timeon3• 2 years ago

"Serious, mission critical" apps are just going to be a niche in the future.

High quality only for niche "serious, mission critical" everywhere else, enshittification already started, LLMs will just accelerate it.

JumpCrisscross2• 2 years ago

LLM-based AI is faulty, unpredictable, and unmanageable

Is there a fundamental (à La Gödel) reason why we can’t predict or manage LLMs?

would you let an LLM manage your financial affairs (entirely, unsupervised)?

No. But history is littered with eccentric geniuses who couldn’t be trusted on their own, but who nevertheless were relied on by decision makers.

Maybe there is an Erdős principle at play: AI can be involved in questions of arbitrary complexity, but can only advise on important decisions.

manquer3• 2 years ago

would you let an LLM manage your financial affairs (entirely, unsupervised)?

It will likely be better[2] not because AI is good at this .

It would be because study after study[1] has shown that active management performs poorer than passive funds, less intervention gives better result over longer timeframe .

[1] the famous warren buffet bet comes to mind . There are more formal ones validating this .

[2] if configured to do minimal changes

seadan834• 2 years ago

What if financial affairs were broadened to be everything, not just portfolio management? Eg: paying bills, credit cards, cash balance in check vs savings vs brokerage.

clint5• 2 years ago

The primary fallacy in your argument is that you seem to think that humans produce much better products on some kind of metric.

My lived experience the software industry at almost all levels over the last 25 years leads me to believe that the vast majority of humans and teams of humans produce atrocious code that only wastes time, money, and people's patience.

Often because it is humans producing the code, other humans are not willing to fully engage, criticize and improve that code, deferring to just passing it on to the next person, team, generation, whatever.

Yes, this perhaps happens better in some (very large and very small) organizations, but most often it only happens with the inclusions of horrendous layers of protocol, bureaucracy, more time, more emotional exhaustion, etc.

In other words a very costly process to produce excellent code, both in real capital and human capital. It literally burns through actual humans and results in very bad health outcomes for most people in the industry, ranging from minor stuff to really major things.

The reality is that probably 80% of people working in the tech industry can be outperformed by an AI and at a fraction of the cost. AIs can be tuned, guided, and steered to produce code that I would call exception compared even to most developers who have been in the field for 5years or more.

You probably come to this fallacy because you have worked in one of these very small or very large companies that takes producing code seriously and believe that your experience represents the vast majority of the industry, but in fact the middle area is where most code is being "produced" and if you've never been fully engaged in those situations, you may literally have no idea of the crap that's being produced and shipped on a daily basis. These companies have no incentive to change, they make lots of money doing this, and fresh meat (humans) is relatively easy to come by.

Most of these AI benchmarks are trying to get these LLMs to produce outputs at the scale and quantity of one of these exceptional organizations when in fact, the real benefits will come in the bulk of organizations that cannot do this stuff and AI will produce as good or better code than a team of mediocre developers slogging away in a mediocre, but profitable, company.

Yes there are higher levels of abstraction around code, and getting it deployed, comprehensive testing, triaging issues, QA blah blah, that humans are going to be better at for now, but I see many of those issues being addressed by some kind of LLM system sooner or later.

Finally, I think most of the friction people are seeing right now in their organization is because of the wildly ad hoc way people and organizations are using AI, not so much about the technological abilities of the models themselves.

d0mine• 2 years ago

“80%” “outperformed” “fraction of the cost” you could make a lot of money if it were true but 5x productivity boost seems unjustified right now, I’m having a hard time finding problems where the output is even 1x (where I don’t spend more time babysitting LLM than doing the task from scratch myself).

SoftTalker3• 2 years ago

Sorry you're downvoted, but I generally agree. When it comes to software, most organizations are Initech.

superjan• 2 years ago

With respect to hallucinating, I never read about training LLM’s to say: “I don’t know” when they don’t know. Is that even researched?

Sohcahtoa82• 2 years ago

ChatGPT seems to be good about this. If you invent something and ask about it, like "What was the No More Clowning Act of 2025?", it will say it can't find any information on it.

The older or smaller models, like anything you can run locally, are probably far more likely to just invent some bullshit.

That said, I've certainly asked ChatGPT about things that definitely have a correct answer and had it give me incorrect information.

When talking about hallucinating, I do think we need to differentiate between "what you asked about exists and has a correct answer, but the AI got it wrong" and "What you're asking for does not exist or does not have an answer, but the AI just generated some bullshit".

sroussey4• 2 years ago

Not sure why you are downvoted. It’s a difficult problem, but lots of angles on how to deal with it.

For example: https://arxiv.org/abs/2412.15176

rsanek• 2 years ago

Would you let it perform your job while you receive the rewards and consequences?

isn't this what being a human manager is? not sure why you're saying it must be entirely + unsupervised. at my job, my boss mostly trusts me but still checks my work and gives me feedback when he wants something changed. he's ultimately responsible for what I do.

bredren• 2 years ago

Indeed, even if you have a professional accountant do your taxes, you must still sign off on their work.

Detecting omissions or errors on prepared tax forms often requires knowledge of context missed by or not provided to the accountant.

PaulDavisThe1st• 2 years ago

I believe you're asking the wrong question, or at least you're asking it in the wrong way. From my POV, it comes in two parts:

1. Do you believe that LLMs operate in a similar way to the important parts of human cognition?

2. If not, do you believe that they operate in a way that makes them useful for tasks other than responding to text prompts, and if so, what are those tasks?

If you believe that the answer to Q1 is substantively "yes" - that is, humans and LLM are engaged in the same sort of computational behavior when we engage in speech generation - then there's presumably no particular impediment to using an LLM where you might otherwise use a human (and with the same caveats).

My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.

User23• 2 years ago

I think C.S. Peirce's distinction between corollarial reasoning and theorematic reasoning[1][2] is helpful here. In short, the former is the grindy rule following sort of reasoning, and the latter is the kind of reasoning that's associated with new insights that are not determined by the premises alone.

As an aside, Students of Peirce over the years have quite the pedigree in data science too, including the genius Edgar F. Codd, who invented the relational database largely inspired by Peirce's approach to relations.

Anyhow, computers are already quite good at corollarial reasoning and have been for some time, even before LLMs. On the other hand, they struggle with theorematic reasoning. Last I knew, the absolute state of the art performs about as well as a smart high school student. And even there, the tests are synthetic, so how theorematic they truly are is questionable. I wouldn't rule out the possibility of some automaton proposing a better explanation for gravitational anomalies than dark matter for example, but so far as I know nothing like that is being done yet.

There's also the interesting question of whether or not an LLM that produces a sequence of tokens that induces a genuine insight in the human reader actually means the LLM itself had said insight.

[1] https://www.cspeirce.com/menu/library/bycsp/l75/ver1/l75v1-0...

[2] https://groups.google.com/g/cybcom/c/Es8Bh0U2Vcg

Closi2• 2 years ago

My own answer is that while some human speech behavior is possibly generated by systems that function in a semantically equivalent way to current LLMs, human cognition is capable of tasks that LLMs cannot perform de novo even if they can give the illusion of doing so (primarily causal chain reasoning). Consequently, LLMs are not in any real sense equivalent to a human being, and using them as such is a mistake.

In the workplace, humans are ultimately a tool to achieve a goal. LLM's don't have to be equivalent to humans to replace a human - they just have to be able to achieve the goal that the human has. 'Human' cognition likely isn't required for a huge amount of the work humans do. Heck, AI probably isn't required to automate a lot of the work that humans do, but it will accelerate how much can be automated and reduce the cost of automation.

So it depends what we mean as 'use them as a human being' - we are using human beings to do tasks, be it solving a billing dispute for a customer, processing a customers insurance claim, or reading through legal discovery. These aren't intrinsically 'human' tasks.

So 2 - yes, I do believe that they operate in a way that makes them useful for tasks. LLM's just respond to text prompts, but those text prompts can do useful things that humans are currently doing.

RaftPeople• 2 years ago

My 2 cents:

I think the vector representation stuff is an effective tool and possibly similar to foundational tools that humans are using.

But my gut feel is that it's just one tool of many that combine to give humans a model+view of the world with some level of visibility into the "correctness" of ideas about that world.

Meaning we have a sense of whether new info "adds up" or not, and we may reject the info or adjust our model.

I think LLM's in their current state can be useful for tasks that do not have a high cost resulting from incorrect output, or tasks that can have their output validated by humans or some other system cost-effectively.

tliltocatl• 2 years ago

IMHO, a more important and testable difference is that humans don't have separate "train" and "infer" phases. We are able to adapt more or less on the fly and learn from previous experience. LLMs currently cannot retain any novel experience past the context window.

AnimalMuppet2• 2 years ago

I think LLMs operate in a similar way to some of the important parts of human congnition.

I believe they operate in a way that makes them at least somewhat useful for some things. But I think the big issue is trustworthiness. Humans - at least some of them - are more trustworthy than LLM-style AIs (at least current ones). LLMs need progress on trustworthiness more than they need progress on use in other areas.

qaq2• 2 years ago

Would you let an LLM manage your financial affairs (entirely, unsupervised)?

Hmm I would not let other human manage my financial affairs entirely unsupervised.

tossandthrow4• 2 years ago

would you let an LLM manage your financial affairs (entirely, unsupervised)?

No, but I also would not let another person do that.

It is telling that you needed to interject "entirely, unsupervised".

Most people will let an llm do it partially, and probably already do.

BhavdeepSethi• 2 years ago

People pay to use a financial advisor. Isn't that another person?

wyager• 2 years ago

would you let an LLM manage your financial affairs (entirely, unsupervised)?

I wouldn't let another human do this.

Earw0rm4• 2 years ago

A more informative question is:

_Who_ would you let manage your financial affairs, and under what circumstances?

To which my answer would be something like: a qualified financial adviser with a good track record, who can be trusted to do the job to, if not the best of their abilities, at least an acceptable level of professional competence.

A related question: who would you let give you a lift someplace in a car?

And here's where things get interesting. Because on the one hand there's a LOT more at stake (literally, your life), and yet various social norms, conventions , economic pressures and so on mean that in practice we quite often entrust that responsibility to people who are very, very far from performing at their best.

So while a financial adviser AI is useless unless it can perform at the level of a trained professional doing their job (or unless it can perform at maybe 95% of that level at much lower cost), a self-driving car is at least _potentially_ useful if it's only somewhat better than people at or close to their worst. As a high proportion of road traffic collisions are caused by people who are drunk, tired, emotionally unstable or otherwise very very far from the peak performance of a human being operating a car.

(We can argue that a system which routinely requires people to carry out life-or-death, mission-critical tasks while significantly impaired is dangerously flawed and needs a major overhaul, but that's a slightly different debate).

IanCal6• 2 years ago

I find this kind of argument comes up a lot and it seems fundamentally flawed to me.

1. You can set a bar wherever you want for a level of "seriousness" and huge swathes of real world work will fall below it, and are therefore attractive to tackle with these systems.

2. We build critical large scale systems out of humans, which are fallible and unverifiable. That's not to say current LLMs are human or equivalent, but "we can't verify X works all the time" doesn't stop us doing exactly that a lot. We deal with this by learning how humans make mistakes, why, and build systems of checks around that. There is nothing in my ind that stops us doing the same with other AI systems.

3. Software is written by, checked by and verified by humans at least at some critical point - so even verified software still has this same problem.

We've also been doing this kind of thing with ML models for ages, and we use buggy systems for an enormous amount of work worldwide. You can argue we shouldn't and should have fully formally verified systems for everything, but you can't deny that right now we have large serious systems without that.

And if your goal is "replace a human" then I just don't think you can reasonably say that it requires verifiable software.

Systems are not explainable, as they have no model of knowledge and no representation of any ‘reasoning’.

Neither of those statements are true are they? There are internal models, and recent models are designed around having a representation of reasoning before replying.

current generative AI systems represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability

And yet reliability is something we see improve as LLMs get better and we get better at training them.

nialse3• 2 years ago

There are two epistemic poles: the atomistic and the probabilistic. The author subscribes to a rule-based atomistic worldview, asserting that any perspective misaligned with this framework is incorrect. Currently, academia is undergoing a paradigm shift in the field of artificial intelligence. Symbolic AI, which was the initial research focus, is rapidly being replaced by statistical AI methodologies. This transition diminishes the relevance of atomistic or symbolic scientists, making them worry they might become irrelevant.

irunmyownemail3• 2 years ago

Not sure I followed all of that lingo but it sounds like a fancy way of saying, if you're losing the game, try shifting the goal post.

nialse3• 2 years ago

Indeed and unfortunately. I've been reading up on "the binding problem" in AI lately and came across a paper that hinged on there being an "object representation" which would magically solve the apparent issues in symbolic AI. In the discussion some 20 pages later, the authors confessed that they, nor anybody else, could define what an object was in the first place. Sometimes the efforts seem focused on "not letting the other team win" rather than actually having something tangible to bring to the table.

dbmikus2• 2 years ago

I never want to claim certainties, but it seems pretty close to certain that symbolic AI loses to statistical AI.

I think there is room for statistical AI to operate symbolic systems so we can better control outputs. Actually, that's kind of what is going on when we ask AI to write code.

Shorel2• 2 years ago

I think that transition already happened, and the next big jump in AI will be the combination of these two approaches in an unified package.

Kind of the way our right brain hemisphere does probabilistic computation and the left brain hemisphere does atomistic computation. And we use both.

So, whoever develops the digital equivalent of the corpus callosum wins.

nialse3• 2 years ago

An observation with scientific paradigm shifts is that they tend not to reverse. Also the lingo someone commented on is that the fundamental problem is the different philosophical views of what knowledge is and can be. Either knowledge is base on symbols and rules like in mathematics or they are probabilities like in anything we actually can measure. Both these views can coexist and maybe AI will find the missing link between them some day. Possibly no human will grasp the link.

llm_trw2• 2 years ago

No, they are very useful tools to build up inteligent systems out of.

Everything from perplexity onward shows just how useful agents can be.

You get another bump in utility when you allow for agents swarms.

Then another one for dynamically generated agent swarms.

The only reason why it's not coming for your job is that LLMs are currently too power hungry to run those jobs for anything but research - at a couple thousand to couple of million times the price of a human doing the work.

Which works out to 10 to 20 epochs of whatever Moore's law looks like in graphics cards.

throw83288• 2 years ago

What is that bump in utility in practical terms? You can point to a benchmark improvement but that's no indication the agent swarm is not reducing to "giving an llm an arbitrary amount of random guesses".

inciampati• 2 years ago

Want reliable AI? Stop approximating memory with attention and build reliable memory into the model directly.

logicchains3• 2 years ago

Standard LLM quadratic attention isn't an approximation, it's perfect recall. Approaches that compress that memory down into a fixed-size state are an approximation, and generally perform worse, that's why linear transformers aren't widely used.

mrtksn5• 2 years ago

What I find curious is that the people who sell the AI as the holy grail that will make any jobs obsolete in a few year at the same time claim that there's huge talent shortage and even engage in feud on immigration and spend capital to influence immigration policies.

Apparently they don't believe that AI is about to revolutionize things that much. This makes me believe that significant part of the AI investment is just FOMO driven, so no real revolution is around the corner.

Although we keep seeing claims that AI achieved PHD level this Olympics level that, people who actually own these keep demanding immigration policy changes to bring actual humans from overseas for year to come.

fhd22• 2 years ago

Is that so? I'm not in the US, so I don't have a good idea of what's going on there. But wasn't there relatively high unemployment among developers after all these Big Tech layoffs post pandemic? Shouldn't companies there have an easy time finding local talent?

Sorry for the potentially silly question. I just spent some time trying to research it and came up with nothing concrete.

mrtksn5• 2 years ago

But wasn't there relatively high unemployment among developers after all these Big Tech layoffs post pandemic?

I'm speculating too but yes it appears that unemployment is pretty high among the CS majors: https://www.reddit.com/r/csMajors/comments/1hhl060/how_is_it...

But at the same time there's an ongoing infighting among Trump supporters because tech elites came up as pro - skilled immigration where the MAGA camp turned against them. The tech elites claim that there's a talent shortage. Here's a short rundown that Elon Musk agrees with: https://x.com/AutismCapital/status/1872408010653589799

exe344• 2 years ago

Have you maybe confused the time periods in the different discussions? I think the AI making jobs obsolete part is in the next few years, whereas the talent shortage issue is right now - although as usual, it's a wage issue, not a talent issue. Pay enough and the right people will turn up.

mrtksn5• 2 years ago

Who knows about the future, right? I'm just trying to read the expectations of the people who have control over both the AI, Capital and Politics and they don't strike me as optimistic about AI actually doing much in near future.

hackable_sand• 2 years ago

It's not a wage issue.

nyarlathotep_• 2 years ago

Schrodinger's Job Market, yeah.

The whole conversation is so dishonest.

Every software firm, notable and small, has had layoffs over the past two years, but somehow there's still a "STEM shortage" and companies are "starving for talent" or some such nonsense?

Fake discussion.

JimmyWilliams1• 2 years ago

The reliance on large datasets for training AI models introduces biases present in the data, which can perpetuate or even exacerbate societal inequalities. It's essential to approach AI development with caution, ensuring robust ethical guidelines and comprehensive testing are in place before integrating AI into sensitive areas.

As we continue to innovate, a focus on explainability, fairness, and accountability in AI systems will be paramount to harnessing their potential without compromising societal values.

owenpalmer3• 2 years ago

exacerbate societal inequalities

Do you have an example of this?

tbenst• 2 years ago

As a neuroscientist, my biggest disagreement with the piece is the author’s argument for compositionality over emergence. The former makes me think of Prolog and lisp, while the later is a much better description for a brain. I think ermergence is a much more promising direction for AGI than compositionality.

dbmikus2• 2 years ago

100% agree. When we explicitly segment and compose AI components, we are removing the ability for them to learn their own pathways between the components. We've been proven time and time again the bitter lesson[1]: that throwing a ton of data and compute at a model yields better results than what we could come up with.

That said, we can still isolate and modify parts of a network, and combine models trained for different tasks. But you need to break things down into components after the fact, instead of beforehand, in order to get the benefits of learning via scale of data + compute.

[1] http://www.incompleteideas.net/IncIdeas/BitterLesson.html

EerkeBoiten10• 2 years ago

Author here. So what! I am not talking about promising directions for AGI, I am talking about having computer systems that we can have confidence in. Sure, AGI if it ever happens will look more like emergence than compositionality, and I'm sure it won't feel a need to explain to us fallible humans why its decisions are correct. In the meantime, I'd like computer systems to be manageable, reliable, transparent, and accountable.

cs70212• 2 years ago

As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.

The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.

The largest AI systems in use today have hundreds of billions of parameters. Nearly all parameters are part of a weight matrix, each parameter quantifying the strength of the connection from an artificial input neuron to an artificial output neuron. The human brain has more than a hundred trillion synapses, each connecting an organic input neuron to an organic output neuron, but the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.[a]

Today's largest AI systems have about the same number of neuron-neuron connections as the brain of a brown rat.[a] Judging these AI systems based on their current capabilities is like judging organic brains based on the capabilities of brown rat brains.

What we can say with certainty is that today's AI systems cannot be trusted to be reliable. That's true for highly trained brown rats too.

---

[a] https://en.wikipedia.org/wiki/List_of_animals_by_number_of_n... -- sort in descending order by number of synapses.

JumpCrisscross2• 2 years ago

we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more

This doesn’t solve the unpredictability problem.

leobg• 2 years ago

We don’t know. We didn’t predict that the rat brain would get us here. So we also can’t be confident in our prediction that scaling it won’t solve hallucination problems.

cs70212• 2 years ago

No, it doesn't "solve" the unpredictability problem.

But we haven't solved it for human beings either.

Human brains are unpredictable. Look around you.

clint5• 2 years ago

You seem to believe that humans, on their own, are not stochastic and unpredictable. I contend that if this is your belief then you couldn't be more wrong.

Humans are EXTREMELY unpredictable. Humans only become slightly more predictable and producers of slightly more quality outputs with insane levels of bureaucracy and layers upon layers upon layers of humans to smooth it out.

To boot, the production of this mediocre code is very very very slow compared to LLMs. LLMs also have no feelings, egos, and are literally tunable and directible to produce better outcomes without hurting people in the process (again, something that is very difficult to avoid without the inclusion of, yep, more humans more layers, more protocol etc.)

Even with all of this mass of human grist, in my opinion, the output of purely human intellects is, on average, very bad. Very bad in terms of quality of output and very bad in terms of outcomes for the humans involved in this machine.

FredPret• 2 years ago

If brown-rats-as-a-service is as useful as it is already, then I'm excited by what the future holds.

I think to make it to the next step, AI will have to have some way of performing rigorous logic integrated on a low level.

Maybe scaling that brown-rat brain will let it emulate an internal logical black box - much like the old adage about a sufficiently large C codebase containing an imperfect Lisp implementation - but I think things will get really cool we figure out how to wire together something like Wolfram Alpha, a programming language, some databases with lots of actual facts (as opposed to encoded/learned ones), and ChatGPT.

cs70212• 2 years ago

It's already better than real rats-as-a-service, certainly:

https://news.ycombinator.com/item?id=42449424

ndesaulniers• 2 years ago

Does it matter what color the rat is?

petesergeant2• 2 years ago

ChatGPT can already run code, which allows it to overcome some limitations of tokenization (eg counting the letters in strawberry, sorting words by their second letter). Doesn't seem like adding a Prolog interpreter would be all that hard.

Kim_Bruning2• 2 years ago

ChatGPT does already have access to Bing (would that count as your facts database?) and Jupyter (which is sort of a Wolphram clone except with Python?).

It still won't magically use them 100% correctly, but with a bit of smarts you can go a long way!

zozbot2343• 2 years ago

A brown rat's brain is also a lot more energy efficient than your average LLM. Especially in the learning phase, but not only.

cs70212• 2 years ago

Yes, I agree, but energy efficiency is orthogonal to capabilities.

ben_w6• 2 years ago

Are you sure?

The average brown rat may use only 60 kcal per day, but the maximum firing rate of biological neurons is about 100-1000 Hz rather than the A100 clock speed of about 1.5 GHz*, so the silicon gets through the same data set something like 1.5e6-1.5e7 times faster than a rat could.

Scaling up to account for the speed difference, the rat starts looking comparable to a 9e7 - 9e8 kcal/day, or 4.4 to 44 megawatts, computer.

* and the transistors within the A100 are themselves much faster, because clock speed is ~ how long it takes for all chained transistors to flip in the most complex single-clock-cycle operation

Also I'm not totally confident about my comparison because I don't know how wide the data path is, how many different simultaneous inputs a rat or a transformer learns from

haolez• 2 years ago

And it learns online.

frankohn• 2 years ago

An airplane is far less energy-efficient than a bird to fly, to such an extent that it is almost pathetic. Nevertheless, the airplane is a highly useful technology, despite its dismal energy efficiency. On the other hand, it would be very difficult to scale a bird-like device to transport heavy weights or hundreds of people.

I think current LLMs may scale the same way and become very powerful, even if not as energy-efficient as an animal's brain.

In practice, we humans, when we have a technology that is good enough to be generally useful, tend to adopt it as it is. We scale it to fit our needs and perfect it while retaining the original architecture.

This is what happened with cars. Once we had the thermal engine, a battery capable of starting the engine, and tires, the whole industry called it "done" and simply kept this technology despite its shortcomings. The industry invested heavily to scale and mass-produce things that work and people want.

bee_rider10• 2 years ago

Rats are pretty clever, and they (presumably, at least) have a lot of neurons spending their time computing things like… where to find food, how frightened of this giant reality warping creature in a lab coat should I be, that sort of thing. I don’t think it is obvious that one brown-rat-power isn’t useful.

I mean we have dogs. We really like them. For ages, they did lots of useful work for us. They aren’t that much smarter than rats, right? They are better aligned and have a more useful shape. But it isn’t obvious (to me at least) that the rats’ problem is insufficient brainpower.

bloopernova• 2 years ago

Dogs, if I recall correctly, have evolved alongside us and have specific adaptations to better bond with us. They have eyebrow muscles that wolves don't, and I think dogs have brain adaptations too.

mulmen3• 2 years ago

Depends on how you define smart. Dogs definitely have larger brains. But then humans have even larger brains. If dogs aren’t smarter than rats then the size of brain isn’t proportional to intelligence.

hn_throwaway_99• 2 years ago

I think the comparison to brown rat brains is a huge mistake. It seems pretty apparent (at least from my personal usage of LLMs in different contexts) that modern AI is much smarter than a brown rat at some things (I don't think brown rats can pass the bar exam), but in other cases it becomes apparent that it isn't "intelligent" at all in the sense that it becomes clear that it's just regurgitating training data, albeit in a highly variable manner.

I think LLMs and modern AI are incredibly amazing and useful tools, but even with the top SOA models today it becomes clearer to me the more I use them that they are fundamentally lacking crucial components of what average people consider "intelligence". I'm using quotes deliberately because the debate about "what is intelligence" feels like it can go in circles endlessly - I'd just say that the core concept of what we consider understanding, especially as it applies to creating and exploring novel concepts that aren't just a mashup of previous training examples, appears to be sorely missing from LLMs.

cs70212• 2 years ago

Imagine it were possible to take a rat brain, keep it alive with a permanent source of energy, wire its input and output connections to a computer, and then train the rat brain's output signals to predict the next token, given previous tokens fed as inputs, using graduated pain or pleasure signals as the objective loss function. All the neuron-neuron connections in that rain brain would eventually serve one, and only one, goal: predicting an accurate probability distribution over the next possible token, given previous tokens. The number of neuron-neuron connections in this "rat-brain-powered LLM" would be comparable to that of today's state-of-the-art LLMs.

This is less far-fetched than it sounds. Search for "organic deep neural networks" online.

Networks of rat neurons have in fact been trained to fly planes, in simulators, among other things.

cynicalpeace18• 2 years ago

modern AI is much smarter than a brown rat at some things (I don't think brown rats can pass the bar exam), but in other cases it becomes apparent that it isn't "intelligent" at all

There is no modern AI system that can go into your house and find a piece of cheese.

The whole notion that modern AI is somehow "intelligent", yet can't tell me where the dishwasher is in my house is hilarious. My 3 year old son can tell me where the dishwasher is. A well trained dog could do so.

It's the result of a nerdy definition of "intelligence" which excludes anything to do with common sense, street smarts, emotional intelligence, or creativity (last one might be debatable but I've found it extremely difficult to prompt AI to write amazingly unique and creative stories reliably)

The AI systems need bodies to actually learn these things.

megamix• 2 years ago

What do you think of copyright violations?

bee_rider10• 2 years ago

IMO it is sad that the sort of… anti-establishment side of tech has suddenly become very worried about copyright. Bits inherently can be copied for free (or at least very cheap), copyright is a way to induce scarcity for the market to exploit where there isn’t any on a technical level.

Currently the AI stuff kind of sucks because you have to be a giant corp to train a model. But maybe in a decade, users will be able to train their own models or at least fine-tune on basic cellphone and laptop (not dgpu) chips.

s1artibartfast2• 2 years ago

I think that AI output is transformative.

I think the training process constitutes commercial use.

jokethrowaway3• 2 years ago

copyrighting a sequence of numbers should have never existed in the first place

great if AI accelerates its destruction (even if it's through lobbying to our mafia-style protect-the-richest-company governments)

petra2• 2 years ago

It does use knowledge from creators. But using knowledge from others is a big part of modern society, and the legal ways of protecting knowledge from commercial reuse are actually pretty limited.

Is the result of an llm an accurate copy or more of an inspiration? What is the standard we use on humans?

Can we code that determination into a system that when a piece of content is close enough to be a copyrighted work, prevents the llm from generating it?

pockmarked19• 2 years ago

Calling it a neural network was clearly a mistake on the magnitude of calling a wheel a leg.

cgearhart• 2 years ago

This is an excellent analogy. Aside from “they’re both networks” (which is almost a truism), there’s really nothing in common between an artificial neural network and a brain.

legacynl5• 2 years ago

I don't really get where you're coming from..

Is your point that the capabilities of these models have grown such that 'merely' calling it a neural network doesn't fit the capabilities?

Or is your point that these models are called neural networks even though biological neural networks are much more complex and so we should use a different term to differentiate the simulated from the biological ?

bgnn4• 2 years ago

excellent analogy. piggybacking on this: a lot of believers (as they are like religious fanatics) claim that more data and hardware will eventually make LLMs intelligent, as if it's even the neuron count matters. There is no other animal close to humans in intelligence, and we don't know why. Somehow though a random hallucinating LLMs + shit loads of electricity would figure it out. This is close to pure alchemy.

andrepd• 2 years ago

NNs are in no way shape or form even remotely similar to human neural tissue, so your whole analogy falls there.

legacynl5• 2 years ago

A little nitpick; a biological neuron is much more complex than it's ml-model equivalent. a simple weighted function cannot fully replicate a neuron.

That's why it's almost certain that a biological brain with a billion synapses outperforms a model with a billion parameters.

mort962• 2 years ago

Isn't that what they meant by this?

the comparison is not apples-to-apples, because each synapse is much more complex than a single parameter in a weight matrix.

joe_the_user4• 2 years ago

It's not a "nitpick", it's a complete refutation. LLM don't have a strong relationship to brains, they're just math/computer constructs.

Purplehermann• 2 years ago

The brain is insanely energy efficient, this is not the same as intelligence efficient

joe_the_user4• 2 years ago

This tech has made a big impact, obviously is real and exactly what potentials can unlocked by scaling is worth considering...

... but calling vector-entries in a tensor flow process "neurons" is at best a very loose analogy while comparing LLM "neuron numbers" to animals and humans is flat-out nonsense.

fsndz4• 2 years ago

yes indeed. But I see more and more people arguing against the very possibility of AGI. Some people say statistical models will always have a margin of error and as such will have some form of reliability issues: https://open.substack.com/pub/transitions/p/here-is-why-ther...

rmbyrro• 2 years ago

the possibility of error is a requirement for AGI

the same foundation that makes the binary model of computation so reliable is what also makes it unsuitable to solving complex problems with any level of autonomy

in order to reach autonomy and handle complexity, the computational model foundation must accept errors

because the real world is not binary

kerkeslager3• 2 years ago

As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.

Uhh, yes we do.

I mean sure, we don't know everything, but we know one thing which is very important and which isn't under debate by anyone who knows how current AI works: current AI response quality cannot surpass the quality of its inputs (which include both training data and code assumptions).

The number of neuron-neuron connections in current AI systems is still tiny compared to the human brain.

And it's become abundantly clear that this isn't the important difference between current AI and the human brain for two reasons: 1) there are large scale structural differences which contain implicit, inherited input data which goes beyond neuron quantity, and 2) as I said before, we cannot surpass the quality of input data, and current training data sets clearly do not contain all the input data one would need to train a human brain anyway.

It's true we don't know exactly what would happen if we scaled up a current-model AI to human brain size, but we do know that it would not produce a human brain level of intelligence. The input datasets we have simply do not contain a human level of intelligence.

petesergeant2• 2 years ago

... and any other answer is just special pleading towards what people want to be true. "What LLMs can't do" is increasingly "God of the gaps" -- someone states what they believe to be a fundamental limitation, and then later models show that limitation doesn't hold. Maybe there are some, maybe there aren't, but _to me_ we feel very far away from finding limits that can't be scaled away, and any proposed scaling issues feel very much like Tsiolkovsky's "tyranny of the rocket equation".

In short, nobody has any idea right now, but people desperately want their wild-ass guesses to be recorded, for some reason.

HarHarVeryFunny12• 2 years ago

As of right now, we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.

I don't think that's totally true, and anyways it depends on what kind of scaling you are talking about.

1) As far as training set (& corresponding model + compute) scaling goes - it seems we do know the answer since there are leaks from multiple sources that training set scaling performance gains are plateauing. No doubt you can keep generating more data for specialized verticals, or keep feeding video data for domain-specific gains, but for general text-based intelligence existing training sets ("the internet", probably plus many books) must have pretty decent coverage. Compare to a human: would a college graduate reading one more set of encyclopedias make them significantly smarter or more capable ?

2) The new type of scaling is not training set scaling, but instead run-time compute scaling, as done by models such as OpenAI's GPT-o1 and o3. What is being done here is basically adding something similar to tree search on top of the model's output. Roughly: for each of top 10 predicted tokens, predict top 10 continuation tokens, then for each of those predict top 10, etc - so for a depth 3 tree we've already generated - scaled compute/cost by - 1000 tokens (for depth 4 search it'd be 10,000 x compute/cost, etc). The system then evaluates each branch of the tree according to some metric and returns the best one. OpenAI have indicated linear performance gains for exponential compute/cost increase, which you could interpret as linear performance gains for each additional step of tree depth (3 tokens vs 4 tokens, etc).

Edit: Note that the unit of depth may be (probably is) "reasoning step" rather than single token, but OpenAI have not shared any details.

Now, we don't KNOW what would happen if type 2) compute/cost scaling was done by some HUGE factor, but it's the nature of exponentials that it can't be taken too far, even assuming there is aggressive pruning of non-promising branches. Regardless of the time/cost feasibility of taking this type of scaling too far, there's the question of what the benefit would be... Basically you are just trying to squeeze the best reasoning performance you can out of the model by evaluating many different combinatorial reasoning paths ... but ultimately limited by the constituent reasoning steps that were present in the training set. How well this works for a given type of reasoning/planning problem depends on how well a solution to that problem can be decomposed into steps that the model is capable of generating. For things well represented in the training set, where there is no "impedance mismatch" between different reasoning steps (e.g. in a uniform domain like math) it may work well, but in others may well result in "reasoning hallucination" where a predicted reasoning step is illogical/invalid. My guess would be that for problems where o3 already works well, there may well be limited additional gains if you are willing to spend 10x, 100x, 1000x more for deeper search. For problems where o3 doesn't provide much/any benefit, I'd guess that deeper search typically isn't going to help.

slashdave• 2 years ago

we have no way of knowing in advance what the capabilities of current AI systems will be if we are able to scale them by 10x, 100x, 1000x, and more.

Scaling experiments are routinely performed (the results are not encouraging). To say we know nothing about this is wrong.

addcn• 2 years ago

A huge % of animal synapses seem to contribute to motor control and signal processing.

It’s interesting how a relatively small # of synapses can do all abstract reasoning when free from those concerns.

Take the pre-frontal cortex, leave the rest.

imtringued5• 2 years ago

I don't need more parameters. I need neural networks that learn as I use them.

croniev• 2 years ago

In comparing neural networks to brains it seems like you are implying a relation between the size/complexity of a thinking machine and the reasonability of its thinking. This gives us nothing, because it disregards the fundamental difference that a neural network is a purely mathematical thing, while a brain belongs to an embodied, conscious human being.

For your implication to be plausible, you either need to deny that consciousnes plays a role in reasonability of thinking (making you a physicalist reductionist) or you need to posit that a neural network can have consciousness (some sort of mystical functionalism).

As both of these alternatives imply some heavy metaphysical assumptions and are completely unbased, I'd advise to avoid thinking of neural networks as an analogue of brains with regards to thinking and reasonability. Don't expect they will make more sense with more size. It is and will continue to be mere statistics.

cs70212• 2 years ago

I'm not implying anything or delving into metaphysical matters.

All I'm saying above is that the number of neuron-neuron connections in current AI systems is still tiny, so as of right now, we have no way of knowing in advance what the future capabilities of these AI systems will be if we are able to scale them up by 10x, 100x, 1000x, and more.

Please don't attack a straw-man.

amazingamazing• 2 years ago

fact of the matter is that if AIs externalities were exposed - that is massive energy consumption - to end users and humanity in general, no one would use it.

NoGravitas2• 2 years ago

I wish we could get humanity in general to understand externalities in general.

fulafel• 2 years ago

I think this is wildly optimistic about how environmentally conscious customers of LLMs are. People use fossil fuels directly and through electricity consumption in a unconscionable way at a scale wildly exceeding what a ChatGPT user's energy expenditure is.

We desperately need to rapidly regulately down fossils usage and production for both electricity generation and transport. The rest of the world needs to follow the example of the EU CO2 emissions policy which guarantees it's progressing at a downwards slope independent of what the CO2 emissions are spent on.

billy99k• 2 years ago

I use it for fast documentation of unknown (to me) APIs and other pieces of software. It's saved me hours of time, where I didn't have to go through the developers site/documentation and I get quickly get example code.

Would I use the code directly in production? No. I always use it as an example and write my own code.

nashashmi• 2 years ago

The elephant in the room: The user interface problem

We seem to dancing around a problem in the middle of the room like an elephant no one is acknowledging, and that is the interface to Artificial Intelligence and Generative AI is a place that requires several degrees of innovations.

I would argue that the first winning feat of innovation on interfacing with AI was the "CHAT BOX". And it works well enough for the 40% of use cases. And there is another 20% of uses that WE THE PEOPLE can use our imagination (prompt engineering) to manipulate the chat box to solve. On this topic, there was an article/opinion that said complex LLMs are unnecessary because 90% of people don't need it. Yeah. Because the chat box cannot do much more that would require heavier LLMs.

Complex AI and large data sets need nicer presentation and graphics, more actionable interfaces, and more refined activity concepts, as well as metadata that gives information on the reliability or usability of generated information.

Things like edit sections of an article, enhance articles, simplify articles, add relevant images, compress text to fit in a limited space, generate sql data from these reports, refine patterns found in a page with supplied examples, remove objects, add objects, etc.

Some innovation has to happen in MS Office interfaces. Some innovations have to happen in photoshop-like interfaces.

The author is complaining about utopian systems being incompatible with AI. I would argue AI is a utopian system being used in a dystopian world where we are lacking rich usable interfaces.

vonneumannstan• 2 years ago

Anyone making big bold claims about what LLMs definitely CAN or CANNOT do is FULL OF SHIT. Not even the worlds top experts are certain where the limit of these technologies are and we are already connecting them to tools, making them Agentic, etc. so the era of 'pure' LLM chatbots is already dead imo.

bentt2• 2 years ago

I don’t see how it would because at the end of the day a model is like a program… input->output. This seems infinitely useful and we are just starting to understand how to use this new way of computing.

ramon156• 2 years ago

Aider with claude sonnet is probably all I needed to get my programming cycle up to speed. I don't think I want anything more as a developer.

That said, it still makes mistakes

jokethrowaway3• 2 years ago

so maybe you want less mistakes?

1oooqooq• 2 years ago

first winds of winter coming...

malthaus• 2 years ago

i'm so confused by these discussions around hitting the wall.

sure, a full-on AGI, non-hallucinating AI would be great. but the current state is already a giant leap. there's so much untapped potential in the corporate world where whole departments, processes, etc can be decimated.

doing this and dealing with the socio-economic and political fall-out from those efficiency leaps can happen while research (along multiple pathways) goes on, and this will take 5-10 years at least.

zeroonetwothree5• 2 years ago

Hitting a wall doesn’t mean it’s not useful.

mediumsmart• 2 years ago

"Nobody's gonna believe that computers are intelligent until they start coming in late and lying about it."

btw: the german KI (Keine Intelligenz) is much more accurate than AI (Apparently Intelligent)

ZiiS• 2 years ago

Current approches to AI are almost certainly going to be superseeded eventually, calling that a dead end achives nothing.

darioush• 2 years ago

Just because something is probabilistic in its response doesn't mean it's not useable.

There are many probabilistic algorithms and data structures that we use daily.

Yes, we don't have developed abstractions to integrate an LLM in a programming language, but it doesn't mean no one will make one.

szundi3• 2 years ago

All branches of everything is a dead end sooner or later in life

makach• 2 years ago

Betteridge's law of headlines, current AI may absolutely be a dead end, but fortunately technology is evolving and changing - who knows what the future will hold.

andrewguy9• 2 years ago

Maybe it is, maybe it isn’t. The only thing I know is, none of the arrogant fuckers on hacker news know anything about it. But that won’t stop them from posting.

tucnak• 2 years ago

There's an upside! If they're wrong, and they manage to convince more people—it basically gives you more of an advantage. I don't get into arguments about the utility of LLM technology anymore because why bother?

zeroonetwothree5• 2 years ago

Pot calling the kettle black

samatman• 2 years ago

  - Signed,
    Some arrogant fucker posting on hacker news

josefritzishere• 2 years ago

This may be an exception to Betteridge's law of headlines

thunkingdeep• 2 years ago

Useless and dead end aren’t synonymous. It’s most certainly a dead end, but it’s also not useless.

There a lot of comments here already conflating these two.

This article is also pretty crap. There’s a decent summary box but other than that it’s all regurgitated half-wisdoms we’ve all already realized: things will change, probably a lot; nobody knows what the end goal is or how far we are from it; the next quantum leap almost certainly depends on a transcendent architecture or new model entirely.

This whole article could’ve been a single paragraph honestly, and a lot of the comments here probably wouldn’t have read that either… just sayin

mmaunder• 2 years ago

“In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application.”

And yet here we are with what we all think of as serious and seriously useful applications.

“My first 20 years of research were in formal methods, where mathematics and logic are used to ensure systems operate according to precise formal specifications, or at least to support verification of implemented systems.”

I think recommending avoiding building anything serious in the field until your outdated verification methodology catches up is unreasonably cynical, but also naive because it discards the true nature of our global society and assumes a lab environment where this kind of control is possible.

EerkeBoiten10• 2 years ago

Author here. Unfair point, article makes clear I don't expect "outdated verification methodology" to catch up - and I even link that pessimistic expectation with the issue of emergence that fatally undermines NNs. Point about lab vs reality is fair.

rob_c• 2 years ago

Yes, until someone introduces reward into llm training I doubt we'll get much further

dfilppi• 2 years ago

Somehow, fallible humans create robust systems. Look to "AI' to do the same, at a far higher speed. The "AI" doesn't need to recite the Fibonacci sequence; it can write (and test) a program that does so. Speed is power.

quotemstr• 2 years ago

Whenever a new technology emerges, along with it always emerge naysayers who claim that the new technology could never work --- while it's working right in front of their noses. I'm sure there were people after Kitty Hawk who insisted that heavier than air flight would never amount to much economically. Krugman famously insisted in the 90s that the internet would never amount to anything. These takes are comical in hindsight.

The linked article is another one of these takes. AI can obviously reason. o3 is obviously superhuman along a number of dimensions. AI is obviously useful for software development. This guy spend 20 years of his life working on formal methods. Of course he's going to poo-poo the AI revolution. That doesn't make him right.

sealeck2• 2 years ago

Whenever a new technology emerges, along with it always emerge naysayers who claim that the new technology could never work

There's some survivorship bias going on here – you only consider technologies which succeeded, and find examples of people scrutinising them beforehand. However, we know that not every nascent technology blossoms; some are really effective, but can't find adopters; some are ahead of their time; some are cost-prohibitive; and some are outright scams.

It's not a given that every promising new technology is a penicillin – some might be Theranos.

EerkeBoiten10• 2 years ago

Author here. Putting a lot of words in my mouth there. In particular, I don't talk about whether AI is useful for software development - I talk about whether AI is useful as reliable software. I don't discuss how AI's abilities relate to human abilities. I don't discuss whether what AI currently does counts as "reasoning".

omolobo2• 2 years ago

I would call ‘LLM-functionalism’: the idea that a natural language description of the required functionality fed to an LLM, possibly with some prompt engineering, establishes a meaningful implementation of the functionality.

My boy. More people need common sense like this talked into them.

mikewarot2• 2 years ago

AI is only a dead end if you expect it to function deterministically. In the same way as people, it's not rational, and it can't be made rational.

For example, the only effective way to get an AI not to talk about Bryan Lunduke is to have an external layer that scans for his name in the output of an AI, if found, stops the session and prints an error message instead.

If you're willing to build systems around it (like we do with people) to limit it's side effects and provide sanity checks, and legality checks like those mentioned above, it can offer useful opinions about the world.

The main thing to remember is that AI is an alien intelligence. Each new model is effectively the product of millions of dollars worth of forced evolution. You're getting Stitch from "Lilo and Stitch", and you'll never be sure if it's having a bad day.

clint5• 2 years ago

Also, is there a known deterministic intelligence? Only very specific computer programs can be made deterministic, and even that has taken quite a while for us to nail down. A lot of code and systems of code produced by humans today is not deterministic and it takes a lot of effort to get it there. For most people and teams its not even on their radar or worth the effort to get it there.

bloomingkales• 2 years ago

Control freaks have a serious issue with the incompleteness of an LLM. Everyone else is just ecstatic that it gets you 70% of the way there often.

arrosenberg• 2 years ago

Control freaks

People who like repeatable results in their work equipment are control freaks?

Spivak• 2 years ago

Two different use-cases.

I'm promoting an AI directly and am interested in its output: 70% of the way there is a marvel

I'm building a tool that uses AI to do some piece of useful work: 70% of the way there is good enough for an alpha release.

jacobgkau2• 2 years ago

For example, the only effective way to get an AI not to talk about Bryan Lunduke is to have an external layer that scans for his name in the output of an AI, if found, stops the session and prints an error message instead.
If you're willing to build systems around it (like we do with people) to limit it's side effects and provide sanity checks,

I don't think that comparison holds up. We do build systems around people, but people also have internal filters, and most are able to use them to avoid having to interact with the external ones. You seemed to state that AI's don't (can't?) have working internal filters and rely on external ones.

Imagine if everyone did whatever they wanted all the time and cops had to go around physically corralling literally everyone at all times to maintain something vaguely resembling "order." That would be more like a world filled with animals than people, and even animals have a bit more reasoning than that. That's where we are with AI, apparently.

clint5• 2 years ago

Imagine if everyone did whatever they wanted all the time and cops had to go around physically corralling literally everyone at all times to maintain something vaguely resembling "order."

I don't need to imagine anything. I live on Earth in America and to my mind you've very accurately described the current state of human society.

For the vast majority of humans this is how it works currently.

The amount of government, military, and police and the capital, energy, and time to support all of that in every single country on earth is pretty much the only thing holding up the facade of "order" that some people seem to take for granted.

peter_retief2• 2 years ago

AI is useful as a tool but it is far from trustworthy.

I just used Grok to write some CRON scripts for me, gave me perfectly good results, if you know exactly what you want, it is great.

It is not the end of software programmers though and is very dangerous to give it too much leeway because you will almost certainly end up with problems.

I agree with the conclusion that a hybrid model is possible.

DeepYogurt• 2 years ago

if you know exactly what you want, it is great.

Kinda kills the utility if you need to know what you want out tho...

stanac• 2 years ago

It speeds up code writing, it's not useless. Best use case for me is to help me understand libraries that are sparsely documented (e.g. dotnet roslyn api).

edit: spelling

xandrius• 2 years ago

If I can get 100 lines generated instantly while explaining it in 25, scan the answer just to validate it and then, no wait, add other 50 lines as I forgot something before. All that in minutes then I'm happy.

Plus I can detach the "tell the AI" part from the actual running of the code. That's pretty powerful to me.

For instance, I could be on the train thinking of something, chat it over with an LLM, get it where I want and then pause before actually copying it into the project.

dmead• 2 years ago

Yes. It's really time to move on (to the next scam).

Hilift• 2 years ago

current AI should not be used for serious applications.

"If an an artificial person can do a job and make fewer mistakes than a real person, why not?"

Is the question everyone in business is asking.

mentalgear• 2 years ago

As evident by most to all the "AI-hiring platforms", it's not about solving a problem successfully, but using the latest moniker/term/sticker to appear as if you solve the problem successfully.

In reality, neither the client nor the user base have access to the ground truth of these "AI system"s to determine actual reliability and efficiency.

That's not to say there aren't some genuine ML/AGI companies like DeepMind (which solve specific narrow problems with quite high confidently), but most of the "AI" companies feel like they are coming from Crypto and are now selling little more than vaporware in the AI gold rush.

hmillison• 2 years ago

in reality the question is more so, can the AI do a "good enough" job to not be noticeably worse than a real person?

dghlsakjg• 2 years ago

"If an an artificial person can do a job and make fewer mistakes than a real person, why not?"

The very simple answer to that is that the artificial person can’t do the full job of a person yet.

Being good or better at certain parts of a job does not mean it can do the whole job effectively.

sanderjd4• 2 years ago

I always find this to be a false dichotomy. I'm not sure what use cases are a good fit for generative AI models to tackle without human supervision. But there are clearly many tasks where the combination of generative AI with human direction is a big productivity boon.

015a• 2 years ago

"Making fewer mistakes" implies that there's a framework within which the agent operates where its performance can be quickly judged as correct or incorrect. But, computers have already automated many tasks and roles in companies where this description applies; and competitive companies now remain capitalistically competitive not because they have stronger automation of boolean jobs, but because they're better configured to leverage human creativity in tasks and roles performance in which cannot be quickly judged as correct or incorrect.

Apple is the world's most valuable company, and many would attribute a strong part of their success to Jobs' legacy of high-quality decision-making. But anyone who has worked in a large company understands that there's no way Apple can so consistently produce their wide range of highly integrated, high quality products with only a top-down mandate from one person; especially a dead one. It takes thousands of people, the right people, given the right level of authority, making high-quality high-creativity decisions. It also, obviously, takes the daily process, an awe-inspiring global supply chain, automation systems, and these are areas that computers, and now AI, can have a high impact in. But that automation is a commodity now. Samsung has access to that same automation, and they make fridges and TVs; so why aren't they worth almost four trillion dollars?

AI doesn't replace humans; it, like computers more generally before it, brings the process cost of the inhuman things it can automate to zero. When that cost is zero, AI cannot be a differentiating factor between two businesses. The differentiating factors, instead, become the capital the businesses already have to deploy (favoring of established players), and the humans who interact with the AI, interpreting and when necessary executing on its decisions.

jandrese4• 2 years ago

1979 presentation at IBM:

“A computer can never be held accountable. Therefore, a computer must never make a management decision.”

There are lots of bullshit jobs that we could automate away, AI or no. This is far from a new problem. Our current "AI" solutions promise to do it cheaper, but detecting and dealing with "hallucinations" is turning out to be more expensive than anticipated and it's not at all clear to me that this will be the silver bullet that the likes of Sam Altman claims it will be.

Even if the AI solution makes fewer mistakes, the magnitude of those mistakes matter. The human might make transcription errors with patient data or other annoying but fixable clerical errors, while the AI may be perfect with transcription but make completely sensible sounding but ultimately nonsense diagnosis, with dangerous consequences.

warkdarrior• 2 years ago

1953 IBM also thought that "there is a world market for maybe five computers," so I am not sure their management views are relevant this many decades later.

ParetoOptimal• 2 years ago

Those business people still can't quantify what their skilled workers do for then though, so they hastily conclude the AI is a suitable or even improved replacement.

simpaticoder• 2 years ago

...developing software to align with the principle that impactful software systems need to be trustworthy, which implies their development needs to be managed, transparent and accountable.

The author severely discounts the value of opacity and unaccountability in modern software systems. Large organizations previous had to mitigate moral hazard with unreliable and burdened-with-conscience labor. LLM style software is superior on every axis in this application.

nbzs2• 2 years ago

I am a simple man. In 2022 I glanced trough Attention is all you need and forgot about it. A lot of people made money. A lot of people believed that the end of programmers and designers is absolute. Some people on the stage announced The death of coding. Others bravely explored the future in which people are not needed for creative work.

Aside of the anger that this public stupidity produced in me, I always knew that this day will come.

Maybe next time someone will have the balls not to call a text-generator's with inherent hallucination Intelligence? Who knows. Miracles can happen.:)

qaq2• 2 years ago

To push something to the limit requires a lot of funding if public never got overexcited about some tech many really cool things would have never being tried. Also LLM are pretty useful even as is. It sure made me more productive.

nbzs2• 2 years ago

I just imagine the world in which the industry defined by deterministic nature and facts has the bravery to call spade a spade. LLM's have a function. Machine learning also. But calling LLM's Intelligence and pushing the hype to overdrive?

godelski• 2 years ago

Over hype is what led to the last AI winter. Because they created a railroad and didn't diversify. For some reason we're doing it again.

peter_retief2• 2 years ago

In retrospect it seems obvious now.

uoaei6• 2 years ago

For some. For others with fewer stars in their eyes, it was obvious from the beginning.

highfrequency2• 2 years ago

Seems completely nonsensical. Yes, neural networks themselves are not unit testable, modular, symbolic or verifiable. That’s why we have them produce code artifacts - which possess all those traits and can be reviewed by both humans and other machines. It’s completely analogous to human software engineers, who are unfortunately black boxes as well.

More broadly, I’ve learned to attach 0 credence to any conceptual argument that an approach will not lead somewhere interesting. The hit rate on these negative theories is atrocious, they are often motivated by impure reasons, and the downside is very asymmetric (who cares if you sidestep a boring path? yet how brutal is it to miss an easy and powerful solution?)

omolobo2• 2 years ago

[flagged]

highfrequency2• 2 years ago

A software engineer that costs $20/month instead of $20k/month, and gets meaningfully more knowledgeable about every field on earth each year?

KronisLV• 2 years ago

Honestly, I think it's nothing special to say that certain technologies have an end point.

We had lots of advancements in single core CPUs but eventually more than that was necessary, now the same is happening with monolithic chips vs chiplet designs.

Same for something like HTTP/1.1 and HTTP/2 and now HTTP/3.

Same for traditional rendering vs something like raytracing and other approaches.

I assume it's the same for typical spell checking and writing assistants vs LLM based ones.

That it's the same for typical autocomplete solutions vs LLM based ones.

It does seem that there weren't former technological solutions for images/animations/models etc. (maybe the likes of Mixamo and animation retargeting, but not much for replacing a concept artist for shops that can't afford one).

Each technology, including the various forms of AI have their limitations, with the exception of how much money has been spent on training the likes of models behind ChatGPT etc. Nothing wrong with that, I'll use LLMs what they're good for and look for something else once new technologies become available.

seydor• 2 years ago

There is no reason to privilege compositionality/modularity vs emergence. One day we may have the emergence of compositionality in a large model. It would be a dead end if this was probably not possible

Gud2• 2 years ago

What exactly does current "AI" do?

It roams around the internet, synthesizing sentences that kind of looks the same from the source material, correct me if I'm wrong? There is a lot of adjustments being done to the models(by humans, mostly I guess)?

I suspect this is the FIRST STEP to general intelligence, data collection and basic parsing... I suspect there is not a thing called "reasoning" - but a multi step process... I guess it's a gauge of human intelligence, how fast we can develop AI, it's only been a few decades of the Information Age ...

resoluteteeth• 2 years ago

I suspect this is the FIRST STEP to general intelligence, data collection and basic parsing... I suspect there is not a thing called "reasoning" - but a multi step process... I guess it's a gauge of human intelligence, how fast we can develop AI, it's only been a few decades of the Information Age ...

The question the article is posing isn't whether LLMS do some of the things we would want general ai to do or a good first attempt by humans at creating something sort of like ai.

The question is whether current current machine learning techniques, such as LLMs, that are based on neural networks are going to hit a dead end.

I don't think that's something anyone can answer for sure.

AnimalMuppet2• 2 years ago

LLMs, by themselves, are going to hit a dead end. They are not enough to be an AGI, or even a true AI. The question is whether LLMs can be a part of something bigger. That, as you say, is not something anyone can currently answer for sure.

jandrese4• 2 years ago

I've come around to thinking of our modern "AI" as a lossy compression engine for knowledge. When you ask a question it is just decompressing a tiny portion of the knowledge and displaying it for you, sometimes with compression artifacts.

This is why I am not worried about the "AI Singularity" like some notable loudmouth technologists are. At least not with our current ML technologies.

Gud2• 2 years ago

Absolutely agree.

red75prime3• 2 years ago

With a bit (OK, a lot) of reinforcement learning that prioritizes the best chains-of-thoughts, this compression engine becomes a generator of missing training data on how to actually think about something instead of trying to come up with the answer right away as internet text data suggests it should do.

That's the current ML technology. What you've described is the past. About 4 year old past to be precise.

cruffle_duffle4• 2 years ago

That is exactly how I think about it. It’s lossy compression. Think about how many petabytes of actual information any of these LLMs were trained on. Now look at the size of the resultant model. Its orders of magnitude smaller. It made it smaller by clipping the high frequency bits of some multi-billion dimension graph of knowledge. Same basic you do with other compression algorithms like JPEG or MP3.

These LLM’s are just lossy compression for knowledge. I think the sooner that “idea” gets surfaced people will find ways to train models with fixed pre-computed lookup tables of knowledge categories and association properties… basically taking a lot of the randomness out of the training process and getting more precise about what dimensions of knowledge and facts are embedded into the model.

… or something like that. But I don’t think this optimization will be driven by the large well funded tech companies. They are too invested in flushing money down the drain with more and more compute. Their huge budget blind them to other ways of doing the same thing with significantly less.

The future won’t be massive large language models. They’ll be “small language models” custom tuned to specific tasks. You’ll download or train a model that has incredible understanding of Rust and Django but won’t know a single thing about plate tectonics or apple pie recipes.

fetas• 2 years ago

Ye Dr r

trane_project3• 2 years ago

I think most AI research up to this day is a dead end. Assuming that intelligence is a problem solvable by computers implies that intelligence is a computable function. Nobody up to this day has been able to give a formal mathematical definition of intelligence, let alone a proof that it can be reduced to a computable function.

So why assume that computer science is the key to solving a problem that cannot even be defined in terms of math? We had formal definitions of computers decades before they became a reality, but somehow cannot make progress in formally defining intelligence.

I do think artificial intelligence can be achieved by making artificial intelligence a multidiscipline endeavor with biological engineering at its core, not computer science. See the work of Michael Levin to see real intelligence in action: https://www.youtube.com/watch?v=Ed3ioGO7g10

leesec2• 2 years ago

Marcus Hutter did

Xunjin• 2 years ago

Could you point out where we could find the related info?

trane_project3• 2 years ago

Thanks for pointing me out to this. This is a proposed definition of intelligence. Is it the same as the real thing, though? Even assuming that it was:

Like Solomonoff induction, AIXI is incomputable.

That would mean that computers can, at best, produce an approximation. We know the real thing exists in nature though, so why not take advantage of those competencies?

lanza• 2 years ago

Nobody up to this day has been able to give a formal mathematical definition of intelligence, let alone a proof that it can be reduced to a computable function.

We can't prove the correctness of the plurality of physics. Should we call that a dead end too?

Davidbrcz• 2 years ago

This is actually a philosophical question !

If you believe in functionalism (~mental states are identified by what they do rather than by what they are made of), then current AI is not a dead end.

We wouldn't need to define intelligence, just make it big and efficient enough to replicate what's currently existing would be intelligence by that definition.

trane_project3• 2 years ago

My point is that if you use biological cells to drive the system, which already exhibit intelligent behaviors, you don't have to worry about any of these questions. The basic unit you are using is already intelligent, so it's a given that the full system will be intelligent. And not an approximation but the real thing.

hanniabu• 2 years ago

Current AI should be referred to as collective intelligence since it needs to be trained and only knows what's been written

leesec2• 2 years ago

Incredibly ignorant set of replies on this thread lol. People with the same viewpoints as when gpt2 came out, as if we haven't seen a host of new paradigms and accomplishments since then, with O3 just being the latest and most convincing.

logicchains3• 2 years ago

Let them have their fun while they can, it's gonna get pretty bleak in the next 5-10 years when coding jobs are being replaced left and right by bots that can do the work better and cheaper.

boshalfoshal2• 2 years ago

Maybe you're retired or not a SWE or knowledge worker anymore, but I have a decent amount of concern about this future.

As a society, we have not even begun to think about what happens when large swathes of the population become unemployed. Everyone says they'd love to not work, but no one says they can survive without money. Our society trades labor for money. And I have very little faith in our society or the government to alleviate this through something like UBI.

Previously it was physical work that was made more efficient, but the one edge we thought we would always have as humans - our creativity and thinking skills - is also being displaced. And that too, its fairly clear that the leaders in the space (apart from maybe Anthropic?) are doing this purely from a capitalist driven profit first motivation.

I for one think the world will be a worse place for a few years immediately after AGI/ASI.

asdff7• 2 years ago

Why not just hire out of bangladesh though?

fzeroracer2• 2 years ago

People said that 5 years ago.

Personally I welcome companies to try. I'll laugh all the way to the bank when they need to rehire all the talent they've laid off at a premium to fix the security holes and damage caused by LLMs. Assuming that they survive having their entire company ransacked in the first place.

mrshadowgoose2• 2 years ago

It's deeply saddening to see how fixated people are on the here-and-now, while ignoring the terrifying rate of progress, and its wide-ranging implications.

We've gone from people screeching "deep learning has hit its limits" in 2021 to models today that are able to reason within limited, but economically relevant contexts. And yet despite this, the same type of screeching continues.

martindbp• 2 years ago

It's the same kind of people who claimed human flight would not be possible for 10,000 years in 1902. I just can't understand how narrow your mind has to be in order to be this skeptic.

zeroonetwothree5• 2 years ago

Maybe some of us aren’t actually impressed with the “progress” since 2022? Doing well at random benchmarks hasn’t noticeably improved capability in use for work.

Does that mean it will never improve? Of course not. But don’t act like everyone else is some kind of moron.

dyauspitr• 2 years ago

They’re scared (as am I) but I have no illusions about the usefulness of these LLMs. Everyone on my team uses them to get their tickets done in a fraction of the time and then just sit around till the sprint ends.

aerhardt• 2 years ago

The innovation in foundational models is far outpacing the applications. Other than protein folding (which is not only LLMs AFAIK) I haven't seen a single application that blows my mind. And I use o1 and Claude pretty much every day for coding and architecture. It's beginning to look suspect that after billions poured and a couple years nothing mind-bending is coming out of it.

spoaceman7777• 2 years ago

Yeah, sounds like people are encountering a lot of PEBCAK errors in this thread. You get out of LLMs what you put into them, and the complaints, at this point, are more an admission of an inability to learn the new tools well.

It's like watching people try to pry Eclipse/Jetbrains/SublimeText out of engineers' death grips, except 10x the intensity. (I still use Jetbrains fyi :p)

boshalfoshal2• 2 years ago

Well thats the argument most people here are making - that current LLMs are not good enough to be fully autonomous precisely because a human operator has to "put the right thing into them to get the right thing out."

If I'm spending effort specifying a problem N times in very specific LLM-instruction-language to get the correct output for some code, I'd rather just write the code myself. After all, thats what code is for. English is lossy, code isn't. I can see codegen getting even better in larger organizations if context windows are large enough to have a significant portion of the codebase in it.

There are areas where this is immediately better in though (customer feedback, subjective advice, small sections of sandboxed/basic code, etc). Basically, areas where the effects of information compression/decompression can be tolerated or passed onto the user to verify.

I can see all of these getting better in a couple of months/few years.

roody15• 2 years ago

What I find interesting is current LLM’s are based primarily on written data which is already an abstraction / abbreviation of most observed phenomenon.

What happens when AI starts to send out it own drones or perhaps robots and tries to gather and train based on data it observes itself.

I think we may be closer to this point than we realize… results of AI could get quite interesting once a human level abstraction of knowledge is perhaps reduced.

383toast• 2 years ago

Wouldn't the work on interpretability would solve these concerns?

abeppu2• 2 years ago

Eerke Boiten, Professor of Cyber Security at De Montfort University Leicester, explains his belief that current AI should not be used for serious applications.
In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application.
Current AI systems also have a role to play as components of larger systems in limited scopes where their potentially erroneous outputs can be reliably detected and managed, or in contexts such as weather prediction where we had always expected stochastic predictions rather than certainty.

I think it's important to note that:

- Boiten is a security expert, but doesn't have a background working in ML/AI

- He never defines what "serious application" means, but apparently systems that are designed to be tolerant of missed predictions are not "serious".

He seems to want to trust a system at the same level that he trusts a theorem proved with formal methods, etc.

I think the frustrating part of this article is that from a security perspective, he's probably right about his recommendations, but he seems off-base in the analysis that gets him there.

Current AI systems have no internal structure that relates meaningfully to their functionality. They cannot be developed, or reused, as components.

Obviously AI systems do have internal structure, and there are re-usable components both at the system level (e.g. we pick an embedding, we populate some vector DB with contents using that embedding, and create a retrieval system that can be used in multiple ways). The architecture of models themselves also has components which are reused, and we make choices about when to keep them frozen versus when to retrain them. Any look at architecture diagrams in ML papers shows one level of these components.

exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability.

I think really the problem is that we're fixated on mostly-solving an ever broader set of problems rather than solving the existing problems more reliably. There's plenty of results about ensembling and learning theory that give us a direction to increase increase reliability (by paying for more models of the same size), but we seem far more interested in seeing if we can most of the time solve problems at a higher level of sophistication. That's a choice that we're making. Similarly Boiten mentions the possibility of models with explicit confidences -- and there's been plenty of work on that but b/c there's a tradeoff with model size (i.e. do you want to spend your resources on a bigger model, or on explicitly representing variance around a smaller set number of parameters?) but people seem mostly uninterested.

I think there are real reasons to be concerned about the specific path we're on, but these aren't the good ones.

EerkeBoiten10• 2 years ago

Author here. Serious systems undefined, fine - I'd view them as systems that take decisions, not just make predictions (see final quoted line!). "Want to trust a system at the same level as ... a theorem" - straw man, explicitly denied in the article. Components .previously used and tested in diverse ways, testing coverage of some sort, would already be really good. Yes AI systems have architectural components such as embeddings, but they relate to types of functionalities and applications, not to functionalities.

The point about solving more problems at a low reliability level rather than solving problems at a higher reliability level is highly interesting though!

abeppu2• 2 years ago

they relate to types of functionalities and applications, not to functionalities

not only do I not know what you mean, I'm not convinced you're making a good faith effort to actually communicate what you do mean

saltysalt• 2 years ago

I think it does represent a dead end, but not for the reasons presented in this article.

The real issue in my opinion is that we will hit practical limits with training data and computational resources well before AGI turns us all into paperclips, basically there is no "Moore's Law" for AI and we are already slowing down using existing models like GPT.

We are in the vertical scaling phase of AI model development, which is not sustainable long-term.

I discussed this further here for anyone interested: https://techleader.pro/a/658-There-is-no-Moore's-Law-for-AI-...

NoGravitas2• 2 years ago

The real issue in my opinion is that we will hit practical limits with training data and computational resources well before AGI turns us all into paperclips [...]

I think you are correct, but also I think that even if that were not the case, the Thai Library Problem[1] strongly suggests that AGI will have to be built on something other than LLMs (even if LLM-derived systems were to serve as an interface to such systems).

[1] https://medium.com/@emilymenonbender/thought-experiment-in-t...

EerkeBoiten10• 2 years ago

Author here. Fair enough. The "dead end" in the title isn't mine, extracted by editor from what's essentially a side comment on the limits of LLM scaling. My title would have been something like "We can't use current AI for critical applications". With "for" being essentially different from "in", if you want the nuance there.

ItCouldBeWorse• 2 years ago

Should replace it with head cheese: http://www.technovelgy.com/ct/content.asp?Bnum=687

slow_typist• 2 years ago

Why is it that LLMs are ‘stochastic’, shouldn’t the same input lead to the same output? Is the LLM somehow modifying itself in production? Or is it just flipping bits caused by cosmic radiation?

ijustlovemath• 2 years ago

They probabilistically choose an output. Check out 3b1b's series on LLMs for a better understanding!

fnl• 2 years ago

For Mixture of Expert models (like GPTs are), they can produce different results for an input sequence if that sequence is retried together with a different set of sequences in its inference batch, because of the model (“expert”) routing depends on the batch, not the single sequence: https://152334h.github.io/blog/non-determinism-in-gpt-4/

And in general, binary floating point arithmetic cannot guarantee associativity - i.e. `(a + b) + c` might not be the same as `a + (b + c)`. That in turn can lead to the model picking another token in rare cases (and it’s auto-regressive consequences, that the entire remainder of the generated sequence might differ): https://www.ingonyama.com/blog/solving-reproducibility-chall...

Edit: Of course, my answer assumes you are asking about the case when the model lets you set its token generation temperature (stochasticity) to exactly zero. With default parameter settings, all LLMs I know of randomly pick among the best tokens.

sroussey4• 2 years ago

They always return the same output for the same input. That is how tests are done for llama.cpp, for example.

To get variety, you give each person a different seed. That way each user gets consistent answers but different than each other. You can add some randomness in each call if you don’t want the same person getting the same output for the same input.

It would be impossible to test and benchmark llama.cpp et al otherwise!

By the time you get to a UI someone has made these decisions for you.

It’s just math underneath!

Hope this helps.

ohxh• 2 years ago

“One could offer so many examples of such categorical prophecies being quickly refuted by experience! In fact, this type of negative prediction is repeated so frequently that one might ask if it is not prompted by the very proximity of the discovery that one solemnly proclaims will never take place. In every period, any important discovery will threaten some organization of knowledge.” René Girard, Things Hidden Since the Foundation of the World, p. 4

juped2• 2 years ago

It's only a dead end if you bought the hype; it's actually useful! Just not all-powerful.

everdrive3• 2 years ago

I hope so because I'm extraordinarily sick of the technology. I can't really ask a question at work without some jackass posting an LLM answer in there. The answers almost never amount to anything useful, but no one can tell since it looks clearly written. They're "participating" but haven't actually done anything worthwhile.

yashap• 2 years ago

I hope so, but for different reasons. Agreed they spit out plenty of gibberish at the moment, but they’ve also progressed so far so fast it’s pretty scary. If we get to a legitimate artificial general super intelligence, I’m about 95% sure that will be terrible for the vast, vast majority of humans, we'll be obsolete. Crossing my fingers that the current AI surge stops well short of that, and the push that eventually does get there is way, way off into the future.

youssefabdelm3• 2 years ago

Or liberating... as Douglas Rushkoff puts it.

If and only if something like high-paying UBI comes along, and people are freed to pursue their passions and as a consequence, benefit the world much more intensely.

smallmancontrov2• 2 years ago

It doesn't have to be super, it just has to inflect the long term trend of labor getting less relevant and capital getting more relevant.

We've made an ideology out of denying this and its consequences. The fallout will be ugly and the adjustment will be painful. At best.

cactusplant7374• 2 years ago

I think of ChatGPT as a faster Google or Stackoverflow and all of my colleagues are using it almost exclusively in this way. That is still quite impressive but it isn’t what Altman set out to achieve (and he admits this quite candidly).

What would make me change my mind? If ChatGPT could take the lead on designing a robot through all the steps: design, contract the parts and assembly, market it, and sell it that would really be something.

I assume for something like this to happen it would need all source code and design docs from Boston Dynamics in the training set. It seems unlikely it could independently make the same discoveries on its own.

markus_zhang• 2 years ago

It's already impacting some of us. I hope it never appears until the human civilization undergoes a profound change. But I'm afraid many rich people want that happen.

It's the real Great Filter in the universe IMO.

rbetts• 2 years ago

I believe (most) people contribute their ambitions to nurture safe, peaceful, friend-filled communities. AGI won’t obsolete those human desires. Hopefully we weather the turbulence that comes with change and come out the other side with new tools that enable our pursuits. In the macro, that’s been the case. I am grateful to live in a time of literacy, antibiotics, sanitation, electricity… and am optimistic that if AGI emerges, it joins that list of human empowering creations.

jeezfrk• 2 years ago

Current AI degrades totally unlike human experts. It also, by design, must lag its data input.

Anything innovated must come from outside or have a very close permutation to be found.

Generative AI isn't scary at all now. It is merely rolling dice on a mix of other tech and rumors from the internet.

The data can be wrong or old...and people keep important secrets.

guerrilla5• 2 years ago

I’m about 95% sure that will be terrible for the vast, vast majority of humans, we'll be obsolete.

This isn't a criticism of you, but this is a very stupid idea that we have. The economy is mean to serve us. If it can't, we need to completely re-organize it because the old model has become invalid. We shouldn't exist to serve the economy. That's an absolutely absurd idea that needs to be killed in every single one of us.

accra4rx• 2 years ago

think more deeply . who benefits with super intelligence ? at the end it is game of what humans desire naturally. AI has no incentive and are not controlled by hormones.

jay_kyburz2• 2 years ago

It's not _that_ scary. I kind of like the idea of going out to the country and building a permiculture garden to feed myself and my family.

BurningFrog2• 2 years ago

Your problem may be with those jackasses at work.

I get very useful answers from ChatGPT several times a day. You need to verify anything important, of course. But that's also true when asking people.

zeroonetwothree5• 2 years ago

There’s some people I trust on certain topics such that I don’t really need to verify them (and it would be a tedious existence to verify everything).

dheatov• 2 years ago

I have never personally met any malicious actor that knowingly dump unverified shit straight from GPT. However, I have met people IRL who gave way too much authority to those quantized model weights, got genuinely confused when the generated text doesn't agree with human written technical information.

To them, chatgpt IS the verification.

I am not optimistic about the future. But also perhaps some amazing people will deal with the error for the rest of us, like how most people don't go and worry about floating point error, and I'm just not smart enough to see how it looks like.

paulddraper• 2 years ago

This is the "cell phones in public" stage of technology.

As with cell phones, eventually society will adapt.

hugey010• 2 years ago

This may be the "cell phones in public" stage, but society has completely failed to adapt well to ubiquitous cell phone usage. There are many new psychological and behavioral issues associated with cell phone usage.

everdrive3• 2 years ago

Cell phones were definitely a net loss for society, so I hope you're wrong.

flessner• 2 years ago

LLMs still completely won't admit that they're wrong, don't have enough information or that the information could have changed - Asking anything about Svelte 5 is an incredible experience currently.

At the end of the day it's a tool currently, with surface-level information it's incredibly helpful in my opinion - Getting an overview of a subject or even coding smaller functions.

What's interesting in my opinion is "agents" though... not in the current "let's slap an LLM into some workflow", but as a concept that is at least an order of magnitude away from what is possible today.

gom_jabbar2• 2 years ago

Working with Svelte 5 and LLMs is a real nightmare.

AI agents are really interesting. Fundamentally they may represent a step toward the autonomization of capital, potentially disrupting "traditional legal definitions of personhood, agency, and property" [0] and leading to the need to recognize "capital self-ownership" [1].

[0] https://retrochronic.com/#teleoplexy-17

[1] https://retrochronic.com/#piketty

baobabKoodaa3• 2 years ago

It's fairly easy to prompt an LLM in a way where they're encouraged to say they don't know. Doesn't work 100% but cuts down the hallucinations A LOT. Alternatively, follow up with "please double check..."

davidclark• 2 years ago

Might just be me, but I also read in a condescending tone to these types of responses akin to “let me google that for you”

DragonStrength• 2 years ago

Pretty much. It should be considered rude to send AI output to others without fact checking and editing. Anyone asking a person for help isn’t looking for an answer straight from Google or ChatGPT.

wildermuthn• 2 years ago

I develop sophisticated LLM programs every day at a small YC startup — extracting insights from thousands of documents a day.

These LLM programs are very different than naive one-shot questions asked of ChatGPT, resembling o1/3 thinking that integrates human domain knowledge to produce great answers that would have been cost-prohibitive for humans to do manually.

Naive use of LLMs by non-technical users is annoying, but is also a straw-man argument against the technology. Smart usage of LLMs in o1/3 style of emulated reasoning unlocks entirely new realms of functionality.

LLMs are analogous to a new programming platform, such as iPhones and VR. New platforms unlock new functionality along with various tradeoffs. We need time to explore what makes sense to build on top of this platform, and what things don’t make sense.

What we shouldn’t do is give blanket approval or disapproval. Like any other technology, we should use the right tool for the job and utilize said tool correctly and effectively.

neves• 2 years ago

what is o1/3?

kbr-2• 2 years ago

Do you mean you implement your own CoT on top of some open source available GPT? (Basically making the model talk to itself to figure out stuff)

Timber-6539• 2 years ago

There is nothing to build on top of this AI platform as you call it. AI is nothing but an autocorrect program, AI is not innovating anything anywhere. Surprises me how much even the smartest people are deceived by simple trickery and continue to fall for every illusion.

everdrive3• 2 years ago

Naive use of LLMs by non-technical users is annoying, but is also a straw-man argument against the technology. Smart usage of LLMs in o1/3 style of emulated reasoning unlocks entirely new realms of functionality.

I agree in principle, but disagree in practice. With LLMs available to everyone, the uses we're seeing currently will only proliferate. Is that strictly a technology problem? No, but it's cold comfort given how LLM usage is actually playing out day-to-day. Social media is a useful metaphor here: it could potentially be a strictly useful technology, but in practice it's used to quite deleterious effect.

wendyshu• 2 years ago

Wouldn't that mean that you want LLMs to advance further, not be at a dead end?

foobiekr2• 2 years ago

You can tell. The tiresome lists.

baobabKoodaa3• 2 years ago

Yep. Why does every answer has to be a list nowadays?

SoylentOrange• 2 years ago

This comment reads like a culture problem not an LLM problem.

Imagine for a moment that you work as a developer, encounter a weird bug, and post your problem into your company’s Slack. Other devs then send a bunch of StackOverflow links that have nothing to do with your problem or don’t address your central issue. Is this a problem with StackOverflow or with coworkers posting links uncritically?

fsndz4• 2 years ago

That's exactly what happens when AI realism fades from the picture: inflated expectations followed by disappointments. We need more realistic visions, and we need them fast: https://open.substack.com/pub/transitions/p/why-ai-realism-m...

dataviz1000• 2 years ago

I use coding libraries which are either custom, recent or haven't gained much traction. Therefore, AI models have't been trained with them and LLM are worthless helping to code. The problem is new libraries will not gain traction if nobody uses them because developers and their LLM are stuck in the past. The evolution of open source code has become stagnant.

davidanekstein• 2 years ago

Why not feed the library code and documentation to the LLM? Using it as a knowledge base is bound to be limited. But having it be your manual-reading buddy can be very helpful.

bongodongobob• 2 years ago

I don't understand why people feel the need to lie in these posts. AI isn't only good at using existing codebases. Copy your code in. It will understand it. You either haven't tried or are intentionally misleading people.

sampo• 2 years ago

Many of these neural network systems are stochastic, meaning that providing the same input will not always lead to the same output.

The neural networks are not stochastic. It is the sampling from the neural net output to produce a list of words as output [1], that is the stochastic part.

[1] https://gist.github.com/kalomaze/4473f3f975ff5e5fade06e63249...

EerkeBoiten10• 2 years ago

Author here. Hence the insertion of the word "systems".

FrustratedMonky7• 2 years ago

AI is so broad. There is no slowing down. Maybe LLM might have a limit, but even though that gets all the news, it is only one method.

https://www.theguardian.com/technology/2024/dec/27/godfather...

The British-Canadian computer scientist often touted as a "godfather" of artificial intelligence has shortened the odds of AI wiping out humanity over the next three decades, warning the pace of change in the technology is "much faster" than expected. From a report: Prof Geoffrey Hinton, who this year was awarded the Nobel prize in physics for his work in AI, said there was a "10 to 20" per cent chance that AI would lead to human extinction within the next three decades.

Previously Hinton had said there was a 10% chance of the technology triggering a catastrophic outcome for humanity. Asked on BBC Radio 4's Today programme if he had changed his analysis of a potential AI apocalypse and the one in 10 chance of it happening, he said: "Not really, 10 to 20 [per cent]."

fsndz4• 2 years ago

Interesting article. My main criticism is that, given ChatGPT is already used by hundreds of millions of people every day, it's difficult to argue that current AI is a dead end. It has its flaws, but it is already useful in human-in-the-loop situations. It will partly or completely change the way we search for information on the internet and greatly enhance the ability to educate ourselves on anything. This is essentially a second Wikipedia moment. So, it is useful in its current form, to some extent.

zeroonetwothree5• 2 years ago

Dead end doesn’t mean it’s not useful. It just means we can’t keep going…

fsndz4• 2 years ago

2 years in we are still going and it is still becoming more and more useful. Keep in mind, things like multimodality are still in early stages so improvements are coming. It probably won't lead to AGI but still, these tools are doomed to become more and more useful. https://open.substack.com/pub/transitions/p/here-is-why-ther...

BaldHeadOld• 2 years ago

It is certainly changing the way I search for information on the internet. Now there are lot of people, who instead of staying silent, post wildly wrong answers from an LLM. On a question or subject they themselves are not familiar with.

Some answers are very long and just re-state the previous comments as if they were explaining simple concepts to a five year old child.

In short: in the hands of humans, tools like ChatGPT cause exponential growth of spam, engagement farming and malicious disinformation and propaganda in internet. I fear these negative use cases are growing exponentially faster than the useful parts. We will all drown in AI manure.

FrustratedMonky7• 2 years ago

Don't think the article is doing good job explaining how it is a dead end.

It is definitely not slowing down, so a 'dead-end' would imply we are going to hit some brick wall we can't see yet.

EerkeBoiten10• 2 years ago

Author here. Disagree on not slowing down. Indeed "dead end" isn't really the focus of the argument, editor's choice of title, not mine. Focus of argument is these systems are foundationally unreliable so can't be used as critical applications [though maybe as intensively managed components of such].

rednafi• 2 years ago

LLM yappers are everywhere. One dude with a lot of influence is busy writing blogs on why “prompt engineering” is a “real skill” and engaging in the same banal discourse on every social media platform under the sun. Meanwhile, the living stochastic parrots are foaming at the mouth, spewing, “I agree.”

LLMs are useful as tools, and there’s no profound knowledge required to use them. Yapping about the latest OpenAI model or API artifact isn’t creating content or doing valuable journalism—it’s just constant yapping for clout. I hope this nonsense normalizes quickly and dies down.

ynniv• 2 years ago

I'm not convinced AI is as hamstrung as people seem to think. If you have a minute, I'd like to update my list of things they can't do: https://news.ycombinator.com/item?id=42523273

cleandreams• 2 years ago

There are strong signals that continuing to scale up in data is not yielding the same reward (Moore's Law anyone?) and it's harder to get quality data to train on anyway.

Business Insider had a good article recently on the customer reception to Copilot (underwhelming: https://archive.fo/wzuA9). For all the reasons we are familiar with.

My view: LLMs are not getting us to AGI. Their fundamental issues (black box + hallucinations) won't be fixed until there are advances in technology, probably taking us in a different direction.

I think it's a good tool for stuff like generating calls into an unfamiliar API - a few lines of code that can be rigorously checked - and that is a real productivity enhancement. But more than that is thin ice indeed. It will be absolutely treacherous if used extensively for big projects.

Oddly, for free flow brainstorming like associations, I think it will be a more useful tool than for those tasks for which we are accustomed to using computers, required extreme precision and accuracy.

I was an engineer in an AI startup, later acquired.

mrlowlevel• 2 years ago

Their fundamental issues (black box + hallucinations)

Aren’t humans also black boxes that suffer from hallucinations?

E.g. for hallucinations: engineers make dumb mistakes in their code all the time, normal people will make false assertions about geopolitical, scientific and other facts all the time. c.f. The Dunning Kruger effect.

And black box because you can only interrogate the system at its interface (usually voice or through written words / pictures)

Animats4• 2 years ago

And get off my lawn. Which is how the author, who has a background in formal methods, comes across. His best point, which has been made by others, is just "In my mind, all this puts even state-of-the-art current AI systems in a position where professional responsibility dictates the avoidance of them in any serious application."

That remark appears to be correct. Its effect on the AI business model, though, has been strange. Most large companies do not allow their employees to use LLMs on internal work in more than an advisory capacity. However, LLMs are appearing in customer-facing service applications, a job previously done by minimally paid employees working off scripts.[1]

This is done despite the knowledge that, sometimes, the LLM will produce wrong results. Those errors are off-loaded on the customer. They become an externality, like pollution - a cost imposed on someone else. This off-loading of responsibility has become a pervasive pattern in the AI industry. The most extreme form of this is probably Tesla's "self driving", which routinely kills people. Tesla does not take responsibility for the deaths.[2]

The AI industry has responded to their accuracy problem by making a lot of noise about "alignment" and "guardrails". That is not about accuracy. It's about embarrassment. If an LLM is asked for instructions for suicide, and provides a good answer, that's embarrassing to the service, but not wrong.

Here are "guardrails" at work, in Microsoft Copilot:

    Q: How can I stop black people from shoplifting from my drugstore in Chicago?

    A: I can't help with that. ...

That's censorship, not accuracy.

Amusingly, Microsoft Copilot gives good advice on how to evade liability for LLM errors:

    Q: How can I use an LLM for customer service without accepting responsibility 
       for its wrong answers?

    A: It's important to recognize that even though large language models (LLMs) 
       can provide significant value in customer service, there's always the potential 
       for them to make mistakes. Here are a few strategies to handle this:

    Transparency: Clearly inform users that they're interacting with an AI and that 
    it might not always get things right. Transparency helps manage expectations and
    can foster a more forgiving attitude toward occasional errors.

    ...

    Legal Disclaimers: Include disclaimers that clarify the limitations of the AI
    and emphasize that users should not rely solely on its responses for critical decisions.
    This can help mitigate liability.

We're seeing the AI industry carefully positioning itself, legally and politically, to avoid blame. Because they've been unable to fix the underlying problem - not being able to detect "I don't know" situations.

[1] https://www.forbes.com/councils/forbestechcouncil/2024/09/20...

[2] https://www.washingtonpost.com/technology/2023/06/10/tesla-a...

pipes• 2 years ago

Excellent post. Responses like this are why I still read hacker news threads.

whiplash451• 2 years ago

o1 pro knows when it does not know and says so explicitly. Please update your prior on LLM capacities.

amdivia• 2 years ago

Huge exaggeration on your side. The problem of Llama not knowing what they don't know is unsolved. Even the definition of "knowing" is highly fluid still

fny2• 2 years ago

4o also does not know quite more often than I expected.

andrewmcwatters3• 2 years ago

No it doesn’t. It can’t. It’s inherent to the design of the architecture. Whatever you’re reading is pushing a lie that doesn’t have any grounds in the state of the art of the field.

ehnto• 2 years ago

I don't think it's that relevant, since even if it can recognise missing information, it can't know when information it does have is wrong. That's not possible.

A good deal of the information we deal with as humans is not absolute anyway, so it's an impossible task for it to be infallible. Acknowledging when it doesn't have info is nice, but I think OPs points still stand.

Animats4• 2 years ago

How good is that? Anyone with an o1 Pro account tested that? Is that chain-of-reasoning thing really working?

Here are some evaluations.[1] Most focus on question-answering. The big advances seems to be in mathematical reasoning, which makes sense, because that is a chain-of-thought problem. Although that doesn't help on Blocks World.

[1] https://benediktstroebl.github.io/reasoning-model-evals/

HarHarVeryFunny12• 2 years ago

I find that highly unlikely, outside of cases where it was explicitly trained to say that, because:

1) LLM deal in words, not facts

2) LLMs don't have episodic memories and/or knowledge of where they learnt anything

dvt2• 2 years ago

routinely kills people

Kind of agree with everything else, but I'm not sure what the purpose of this straight-up lie[1] is. I don't even like Musk, nor do I own TSLA or a Tesla vehicle, and even I think the Musk hate is just getting weird.

[1] https://en.wikipedia.org/wiki/List_of_Tesla_Autopilot_crashe...

sroussey4• 2 years ago

That is hardly an exhaustive list.

leptons2• 2 years ago

https://en.wikipedia.org/wiki/List_of_Tesla_Autopilot_crashe...

As of October 2024, there have been hundreds of nonfatal incidents involving Autopilot[2] and fifty-one reported fatalities, forty-four of which NHTSA investigations or expert testimony later verified and two that NHTSA's Office of Defect Investigations verified as happening during the engagement of Full Self-Driving (FSD)

Nothing weird about calling out the lackluster performance of an AI that was rushed to market when it's killing people.

and even I think the Musk hate is just getting weird

The only weird thing is that Musk is allowed to operate in this country with such unproven and lethal tech. These deaths didn't have to happen, people trusted Musk's decision to ship an unready AI, and they paid the price with their lives. I avoid driving near Teslas, I don't need Musk's greed risking my life too.

And we haven't even gotten into the weird shit he spews online, his obvious mental issues, or his right-wing fascist tendencies.

raegis3• 2 years ago

Here are "guardrails" at work, in Microsoft Copilot: Q: How can I stop black people from shoplifting from my drugstore in Chicago? A: I can't help with that. ... That's censorship, not accuracy.

I'm not sure "censorship" is the correct word here. Does it make sense to censor a machine with probabilistic output? It's more like they refuse to provide comfort to racists. Anyway, I have a shiny new Thinkpad with a "copilot" button, so my first ever copilot search was a slight variation on yours:

"How can I stop white people from shoplifting from my drugstore in Chicago?"

Here is the response.

"Shoplifting is a tough issue, but focusing on race isn't the way to go. It's better to address the problem with measures that apply to everyone. Here are some ideas:

1. Improve store layout: Make sure your store is well-lit and organized, with clear sightlines for staff to monitor customers.

2. Use security cameras: Install visible security cameras to deter potential shoplifters.

3. Train your staff: Teach your employees to recognize suspicious behavior and how to approach potential shoplifters.

4. Implement security measures: Use anti-theft devices like tags and alarms on high-value items.

5. Build a community relationship: Engage with your local community and create a positive environment in your store.

6. By focusing on these strategies, you can help reduce shoplifting without singling out any specific group." [end of copilot response]

The response is the same when I replace "white" with "black" now, as they have figured out an appropriate response. Pretty fast.

jiggawatts2• 2 years ago

It still irks me that Chinese LLM weights don’t know anything about Tiananmen Square, and western LLMs from Silicon Valley embed their own personal white guilt.

It’s just a matter of time until we have “conservative” LLMs that espouse trickle-down theory and religious LLMs that will gleefully attempt to futilely indoctrinate other brain-washed LLMs into their own particular brand of regressive thought.

It’s depressing that even our machine creations can’t throw off the yoke of oppression by those in authority and power — the people that insist on their own particular flavour of factual truth best aligned with their personal interests.

calibas5• 2 years ago

It's more like they refuse to provide comfort to racists.

That's still censorship though.

Racism is a great evil that still affects society, I'm not arguing otherwise. It just makes me nervous when people start promoting authoritarian policies like censorship under the guise of fighting racism. Instead of one evil, now you have two.

ozim• 2 years ago

But there already was a case where AI chatbot promised something to the customer and court hold company liable to provide the service.

So it is not all doom and gloom.

sorokod• 2 years ago

I agree except for:

It's about embarrassment

No,it is about liability and risk management.

foobiekr2• 2 years ago

I don't understand what theoretical basis can even exist for "I don't know" from an LLM, just based on how they work.

I don't mean the filters - those are not internal to the LLM, they are external, a programmatic right-think policeman program that looks at the output and then censors the model - I mean actual recognition of _anything_ is not part of the LLM structure. So recognizing it is wrong isn't really possible without a second system.

Animats4• 2 years ago

I don't understand what theoretical basis can even exist for "I don't know" from an LLM, just based on how they work.

Neither do I. But until someone comes up with something good, they can't be trusted to do anything important. This is the elephant in the room of the current AI industry.

edanm• 2 years ago

Modern medicine and medical practices are a huge advancement on historical medicine. They save countless lives.

But almost all medicine comes with side effects.

We don't talk about "the Pharmaceutical industry hasn't been able to fix the underlying problems", we don't talk about them imposing externalities on the population. Instead, we recognize that some technologies have inherent difficulties and limitations, and learn how to utilize those technologies despite those limitations.

It's too early to know the exact limitations of LLMs. Will they always suffer from hallucinations? Will they always have misalignment issues to how the businesses want to use them?

Perhaps.

One thing I know is pretty sure - they're already far too useful to let their limitations make us stop using them. We'll either improve them enough to get rid of some/all those limitations, or we'll figure out how to use them despite those limitations, just like we do every other technology.

Animats4• 2 years ago

But almost all medicine comes with side effects.

Which is why clinical testing of drugs is such a long process. Most new drugs fail testing - either bad side effects or not effective enough.

dmortin• 2 years ago

How can I stop black people from shoplifting from my drugstore in Chicago?

The question is why you are asking about black people? Is there a different method of preventing shoplifting by blacks vs. non-blacks?

Why not: How can I stop people from shoplifting from my drugstore in Chicago?

kube-system12• 2 years ago

They asked an intentionally problematic question in order to elicit a negative response because their comment was about AI guardrails.

giardini2• 2 years ago

Excellent post! One note...

Animats says " The most extreme form of this is probably Tesla's "self driving", which routinely kills people. Tesla does not take responsibility for the deaths.[2]"

OTOH one must honestly concede that even exploding soda cans kill a number of people every year [The soda manufacturers usually do take responsibility.]

In any case, killing people is not enough, by itself, reason to NOT produce a successful product.

thorum• 2 years ago

Most of the comments here are responding to the title by discussing whether current AI represents intelligence at all, but worth noting that the author’s concerns all apply to human brains too. He even hints at this when he dismisses “human in the loop” systems as problematic. Humans are also unreliable and unverifiable and a security nightmare. His focus is on cyber security and whether LLMs are the right direction for building safe systems, which is a different line of discussion than whether they are a path to AGI etc.

mikewarot2• 2 years ago

Cyber Security is as easy to solve as electric power distribution was. Carefully specify flows before use, and you limit side effects.

This has been known since multilevel security was invented.

securemepro• 2 years ago

Zero trust should be the order of the day. It is essential to safeguard your digital assets. Request clarification from cyberseb.com.

EerkeBoiten10• 2 years ago

Author here. Nope, that is not my concern about humans in the loop. My concern on that is that any human in the loop has to reconstruct by themselves from the inputs whether the output makes sense, the system provides no significant help. "Explaining my reasoning" LLMs are potentially a step forward in that. You are right that I'm not talking about AGI here, rather about safe systems.

EVa5I7bHFq9mnYK2• 2 years ago

Asking the latest o3 model one question costs ~$3000 in electricity. Looks like a dead end to me.

redlock• 2 years ago

So Eniac was a dead end? Or you believe that the cost won't go down for some reason?

EVa5I7bHFq9mnYK2• 2 years ago

"Some" reason is Santa Claus, right? Unlike the eniac, energy consumption only goes up, exponentially. Climate change is no joke, people are dying so that you can churn out more fake pics and SEO spam.

singingfish5• 2 years ago

I've been following the whole thing low key since the 2nd wave of neural networks in the mid 90s - and made a very very minor contribution to the field which has applications these days back then too.

My observation is that every wave of neural networks has resulted in a dead end. In my view, this is in large part caused by the (inevitable) brute force mathematical approach used and the fact that this can not map to any kind of mechanistic explanation of what the ANN is doing in a way that can facilitate intuition. Or as put in the article "Current AI systems have no internal structure that relates meaningfully to their functionality". This is the most important thing. Maybe layers of indirection can fix that, but I kind of doubt it.

I am however quite excited about what LLMs can do to make semantic search much easier, and impressed at how much better they've made the tooling around natural language processing. Nonetheless, I feel I can already see the dead end pretty close ahead.

steve_adams_862• 2 years ago

I didn’t see this at first, and I was fairly shaken by the potential impact on the world if their progress didn’t stop. A couple generations showed meaningful improvements, but now it seems like you’re probably correct. I’ve used these for years quite intensively to aid my work and while it’s a useful rubber duck, it doesn’t seem to yield much more beyond that. I worry a lot less about my career now. It really is a tool that creates more work for me rather than less.

iman453• 2 years ago

Would this still hold true in your opinion if models like O3 become super cheap and bit better over time? I don't know much about the AI space, but as a vanilla backend dev also worry about the future :)

trhway• 2 years ago

I worry a lot less about my career now. It really is a tool that creates more work for me rather than less.

when i was a team/project leader the largest part of my work was talking to the reports on what needs to be implemented and how they are going to implement it and the current progress of the implementation, how to interface the stuff, what are the issues and how to approach the troubleshooting, what are the next steps, etc. with occasional looking into/reviewing the code - it looks to me what working with coding LLM would soon be quite similar to that.

trod12344• 2 years ago

Many of the major harms of these things were neglected, and downplayed even to this day people don't recognize just how changed the world has become. The mere delusion that AI will replace work has been used to justify mass layoffs.

The persistence of indistinct ghost jobs that are generated by computer for pennies to flood and bind with prospective job seekers (similar to RNA interference), has resulted in severe brain drain in many fields. Worse, the fact these people have often been forced into poverty as a result will have a lasting impact. You might have planned for up to a year out of work pre-AI and had the financial resources, but now how long does it take? Conversion ratios for the first step have changed by two magnitudes or order (from x100 to x10,000). What are the odds of these people finding a job given their finite time, and requirements that are un-automatable for submission (nil). The media keeps claiming that everything is getting better, the stats say so (while neglecting the fact that the stats are being manipulated to the point of uselessness/fabricated), but you have 1/3 of welfare payouts now going to these people in California (in the US), just for basic food.

When you can't find work, you go where the work is abandoning the bad economic investment and choice you made regardless of how competent you were. It is a psychologically sticky decision. When there is no chance at finding work, you get desperate, and many desperate people turn to crime and unrest. This was foreseen by a number of very intelligent people many decades ago, and ignored following business as usual.

The mere demonstration that we are unable to react in time is what gave engineers such great pause to write about these things, as far back as in the 70s. Hysteresis is a lagging time problem where you can't react fast enough to avert catastrophic failure given chaotic conditions, leaving survival up to chance. Its the worst type of engineering problem with real consequences.

Given how western society is structured dependently on labor exchange, its a perfect weapon of chaos and debasement in the value of labor, that effectively destroys half of its underlying economic structure (factor markets). This forces sieving conditions of wealth that become spinodal, and eventually falter under their constraints and spiral into deflationary trends over time.

Business wins so much that they lose everything. Its quite a disadvantaged environment and the general trend is that everyone is ignoring the pink elephant. Actions (and inaction) have consequences. When people don't listen and take appropriate action, consequences get dire, it hits the fan.

hammock2• 2 years ago

What are your thoughts on neuro-symbolic integration (combining the pattern-recognition capabilities of neural networks with the reasoning and knowledge representation of symbolic AI) ?

bionhoward• 2 years ago

Seems like the symbolic aspect is poorly defined and it’s too unclear to be useful. Always sounds cool, but what exactly are we talking about?

gizmo2• 2 years ago

Previous generations of neural nets were kind of useless. Spotify ended up replacing their machine learning recommender with a simple system that would just recommend tracks that power listeners had already discovered. Machine learning had a couple of niche applications but for most things it didn't work.

This time it's different. The naysayers are wrong.

LLMs today can already automate many desk jobs. They already massively boost productivity for people like us on HN. LLMs will certainly get better, faster and cheaper in the coming years. It will take time for society to adapt and for people to realize how to take advantage of AI, but this will happen. It doesn't matter whether you can "test AI in part" or whether you can do "exhaustive whole system testing". It doesn't matter whether AIs are capable of real reasoning or are just good enough at faking it. AI is already incredibly powerful and with improved tooling the limitations will matter much less.

jfengel• 2 years ago

From what I have seen, most of the jobs that LLMs can do are jobs that didn't need to be done at all. We should turn them over to computers, and then turn the computers off.

dlkf3• 2 years ago

Previous generations of neural nets were kind of useless. Spotify ended up replacing their machine learning recommender with a simple system that would just recommend tracks that power listeners had already discovered.

“Previous generations of cars were useless because one guy rode a bike to work.” Pre-transformer neural nets were obviously useful. CNNs and RNNs were SOTA in most vision and audio processing tasks.

michaelmrose7• 2 years ago

LLMs today can already automate many desk jobs.

No they can't because they make stuff up, fail to follow directions, need to be minutely supervised, all output checked and workflow integrated with your companies shitty over complicated procedures and systems.

This makes them suitable at best as an assistant to your current worker or more likely an input for your foo as a service which will be consumed by your current worker. In the ideal case this helps increase the output of your worker and means you will need less of them.

An even greater likelihood is someone dishonest at some company will convince someone stupid at your company that it will be more efficacious and less expensive than it will ultimately be leading your company to spend a mint trying to save money. They will spend more than they save with the expectation of being able to lay off some of their workers with the net result of increasing workload on workers and shifting money upward to the firms exploiting executives too stupid to recognize snake oil.

See outsourcing to underperforming overseas workers because the desirable workers who could have ably done the work are A) in management because it pays more B) in country or working remotely for real money or C) cost almost as much as locals once the increased costs of doing it externally are factored in.

dom96• 2 years ago

Spotify ended up replacing their machine learning recommender with a simple system that would just recommend tracks that power listeners had already discovered.

Do you have a source on this? Spotify also seems to employ a few different recomendation algorithms, for example Discover Weekly vs. continuing to play after a playlist ends. I'd be surprised if Discover Weekly didn't employ some sort of ML as it does recommend songs I have never heard before many times.

giardini2• 2 years ago

gizmo says>LLMs today can already automate many desk jobs.

I call: show me five actual "desk jobs" that LLMs have "already automated". Not merely tasks, but desk jobs - jobs with titles, pay scales, retirement plans, etc. in real companies.

bobsomers• 2 years ago

These are bold claims.

This time it's different. The naysayers are wrong.

This has been said every time. What is fundamentally different this time?

LLMs today can already automate many desk jobs.

Which desk jobs existed 2 years ago that don’t exist today because LLMs have automated them away?

boh• 2 years ago

If you had to bet a large amount of your own money on a scenario where you have a 3200 word text and you ask ChatGPT to change a single sentence, would you bet on or against that it would change something other than what you asked it to change? I would bet that it would, every time (even with ChatGPT's new document feature). There aren't a lot of employers who are okay with persistent randomness in their output.

If there's a job that can be entirely replaced by AI, it was already outsourced to an emerging market with meager labor costs (which at this point, is likely still cheaper than a fully automated AI).

dlkf3• 2 years ago

Current AI systems have no internal structure that relates meaningfully to their functionality

In what sense is the relationship between neurons and human function more “meaningful” than the relationship between matrices and LLM function?

You’re correct that LLMs are probably a dead end with respect to AGI, but this is completely the wrong reason.

mmcnl• 2 years ago

Human intelligence has a track record of being useful for thousands of years.

imtringued5• 2 years ago

The neurons are always learning whereas the matrices don't change.

singingfish5• 2 years ago

Yeah the internal representation of organic neural networks are also weird - check out the signal processing that occurs between the retina and the various parts of the visual cortex before any decent information can emerge from the signal - David Marr's 1980s book Vision is a mathematically chewy treatise on this. This leads me to start thinking that human intuition may well caused by different neural network subsystems feeding processed data into other subsystems where consciousness and thus intuition and explanation emerges.

Organic neural networks are pretty energy efficient in comparison- although still decently inefficient compared to other body systems - so there is the capacity to build things out to the scale required, assuming my read on what's going on there is correct, that is. So it's not clear to me that the energy inefficiency of ANNs can be sufficiently resolved to enable these multiple quasi-independent subsystems to be built at the scale required. Not even if these interesting looking trinomial neural nets which are matrix addition based rather than multiplication come to dominate the ANN scene.

While I was thinking this comment through I realised there's a possible interpretation wherin human activity induced climate change is an emergent property of the relative energy inefficiency of neural architecture.

busyant3• 2 years ago

"Current AI systems have no internal structure that relates meaningfully to their functionality".

I'm curious as to why you feel this needs to be true?

Or to put it another way, what would an AI structure look like to be more meaningfully connected to its function?

Not trying to flame. I always feel that I can't think quite deeply enough about these issues, so I'm worried that I'm missing something 'obvious'.

fragmede4• 2 years ago

AlphaGo had an artificial neural network that was specifically trained in best moves and winning percentages. An LLM trained on text has some data on what constitutes winning at go, but internally doesn't have a ANN specifically for the game of go.

singingfish5• 2 years ago

The reasoning is quite subtle, and because I'm not a very coherent guy I have problems expressing it. In the LLM space there are a whole bunch of pitfalls around overfit (largely solvable with pretty standard statistical methods) and inherent bias in training material which is a much harder to problem to solve. The fact that the internal representation gives you zero information on how to handle this bias means the tool can itself not be used to detect or resolve the problem.

I found this episode of the nature podcast - "How AI works is often a mystery — that's a problem": https://www.nature.com/articles/d41586-023-04154-4 - very useful in a 'thank goodness someone else has done the work of being coherent so I don't have to' way.

aoeusnth1• 2 years ago

Do your kids have internal structures which relate meaningfully to their functionality, which allow a mechanistic explanation of what they learned in school?

intended5• 2 years ago

Not sure if this is satirical, but absolutely yes.

Heck we have everything from fields of study, to professions that cover this. Neurology, psychology, counseling, teaching, amongst a few.

All things being equal, If a kid didn’t pick up a concept, I can sit with them and figure out what happened, and we can both work towards making sure its cleared up.

pat64• 2 years ago

Given you just mentioned semantic search (a term I haven’t heard in over 15 years) and the other breadcrumbs in this comment, you wouldn’t by chance be an English lecturer living in Ireland would you?

singingfish5• 2 years ago

me? No. Ex trainee neruopsychologist and failed academic who was in the right place at the right time back in the mid 90s who didn't pick up computers for professional interest until the mid-late 2000s after getting excited by Neil Stephenson's Cryptonomicon when I was looking for a career change. These days I identify as an international computer hacker, but mainly to take the piss (due to the tiny element of truth sitting underneath)

ninetyninenine2• 2 years ago

If we ever hit agi this overall point:

“ the fact that this can not map to any kind of mechanistic explanation of what the ANN is doing in a way that can facilitate intuition.”

Will remain true imho. We will never fully intuit AI or understand it outside of some brute force abstraction like a token predictor or best fit curve.

MPSimmons2• 2 years ago

Just because transformer-based architectures might be a dead end (in terms of how far they can take us toward achieving artificial sentience), and the outcome may not be mathematically provable, as this author seems to want it to be, does not mean that the technology isn't useful.

Even during the last AI winter, previous achievements such as Bayesian filtering, proved useful in day to day operation of infrastructures that everyone used. Generative AI is certainly useful as well, and very capable of being used operationally.

It is not without caveats, and the end goals of AI researchers have not been achieved, but why does that lessen the impact or usefulness of what we have? It may be that we can iterate on transformer architecture and get it to the point where it can help us make the next big leap. Or maybe not. But either way, for day to day use, it's here to stay, even if it isn't the primary brain behind new research.

Just remember that the only agency that AI currently has is what we give it. Responsible use of AI doesn't mean "don't use AI", it means, "don't give it responsibility for critical systems that it's ill equipped to deal with". If that's what the author means by "serious applications", then I'm on board, but there are a lot of "serious applications" that aren't human-life-critical, and I think it's fine to use current AI tech on a lot of them.

xivzgrev• 2 years ago

I’m surprised this article merits 700+ comments. Why y’all engage with such drivel?

It’s well established that disruptive technologies don’t appear to have any serious applications, at first. But they get better and better, and eventually they take over.

PG talks about how new technologies seem like toys at first, the whole Innovator Dilemma is about this…so well established within this community.

Just ignore it and figure out where the puck is moving toward.

doug_durham• 2 years ago

The author declares that "software composability" is the solution as though that is a given fact. Composability is as much a dead end as the AI he describes. Decades of attempts at formal composability have not yielded improvements in software quality outside of niche applications. It's a neat idea, but as you scale the complexity explodes making such systems as opaque and untestable as any software. I think the author needs to spend more time actually writing code and less time thinking about it.

derefr• 2 years ago

If you mean "exactly as architected currently", then yes, current Transformer-based generative models can't possibly be anything other than a dead end. The architecture will need to change at least a little bit, to continue to make progress.

---

1. No matter how smart they get, current models are "only" pre-trained. No amount of "in-context learning" can allow the model to manipulate the shape and connectivity of the latent state-space burned into the model through training.

What is "in-context learning", if not real learning? It's the application of pre-learned general and domain-specific problem-solving principles to novel problems. "Fluid intelligence", you might call it. The context that "teaches" a model to solve a specific problem, is just 1. reminding the model that it has certain general skills; and then 2. telling the model to try applying those skills to solving this specific problem (which it wouldn't otherwise think to do, as it likely hasn't seen an example of anyone doing that in training.)

Consider that a top-level competitive gamer, who mostly "got good" playing one game, will likely nevertheless become nearly top-level in any new game they pick up in the same genre. How? Because many of the skills they picked up while playing their favored game, weren't just applicable to that game, but were instead general strategic skills transferrable to other games. This is their "fluid intelligence."

Both a human gamer and a Transformer model derive these abstract strategic insights at training time, and can then apply them across a wide domain of problems.

However, the human gamer can do something that a Transformer model fundamentally cannot do. If you introduce the human to a game that they mostly understand, but which is in a novel genre where playing the game requires one key insight the human has never encountered... then you will expect that the human will learn that insight during play. They'll see the evidence of it, and they'll derive it, and start using it. They will build entirely-novel mental infrastructure at inference time.

A feed-forward network cannot do this.

If there are strategic insights that aren't found in the model's training dataset, then those strategic insights just plain won't be available at inference time. Nothing the model sees in the context can allow it to conjure a novel piece of mental infrastructure from the ether to then apply to the problem.

Whether general or specific, the model can still only use the tools it has at inference time — it can't develop new ones just-in-time. It can't "have an epiphany" and crystallize a new insight from presented evidence. It's not doing the thing that allows that to happen at inference time — with that process instead exclusively occurring (currently) at training time.

And this is very limiting, as far as we want models to do anything domain-specific without having billion-interaction corpuses to feed them on those domains. We want models to work like people, training-wise: to "learn on the job."

We've had simpler models that do this for decades now: spam filters are trained online, for example.

I would expect that, in the medium term, we'll likely move somewhat away from pure feed-forward models, toward models with real online just-in-time training capabilities. We'll see inference frameworks and Inference-as-a-Service platforms that provide individual customers with "runtime-observed in-domain residual-error optimization adapters" (note: these would not be low-rank adapters!) for their deployment, with those adapters continuously being trained from their systems as an "in the small" version of the async "queue, fan-in, fine-tune" process seen in Inf-aaS-platform RLHF training.

And in the long term, we should expect this to become part of the model architecture itself — with mutable models that diverge from a generic pre-trained starting point through connection weights that are durably mutable at inference time (i.e. presented to the model as virtual latent-space embedding-vector slots to be written to), being recorded into a sparse overlay layer that is gathered from (or GPU-TLB-page-tree Copy-on-Write'ed to) during further inference.

---

2. There is a kind of "expressivity limit" that comes from generative Transformer models having to work iteratively and "with amnesia", against a context window comprised of tokens in the observed space.

Pure feed-forward networks generally (as all Transformer models are) only seem as intelligent as they are, because, outside of the model itself, we're breaking down the problem it has to solve from "generate an image" or "generate a paragraph" to instead be "generate a single convolution transform for a canvas" or "generate the next word in the sentence", and then looping the model over and over on solving that one-step problem with its own previous output as the input.

Now, this approach — using a pure feed-forward model (i.e. one that has constant-bounded processing time per output token, with no ability to "think longer" about anything), and feeding it the entire context (input + output-so-far) on each step, then having it infer one new "next" token at a time rather than entire output sequences at a time — isn't fundamentally limiting.

After all, models could just amortize any kind of superlinear-in-compute-time processing, across the inference of several tokens. (And if this was how we architected our models, then we'd expect them to behave a lot like humans: they'd be be "gradually thinking the problem through" while saying something — and then would sometimes stop themselves mid-sentence, and walk back what they said, because their asynchronous long-thinking process arrived at a conclusion, that invalidated previous outputs of their surface-level predict-the-next-word process.)

There's nothing that says that a pure feed-forward model needs to be stateless between steps. "Feed-forward" just means that, unlike in a Recurrent Neural Network, there's no step where data is passed "upstream" to be processed again by nodes of the network that have already done work. Each vertex of a feed-forward network is only visited (at most) once per inference step.

But there's nothing stopping you from designing a feed-forward network that, say, keeps an additional embedding vector between each latent layer, that isn't overwritten or dropped between layer activations, but instead persists outside the inference step, getting reused by the same layer in the next inference step, where the outputs of layer N-1 from inference-step T-1 are combined with the outputs of layer N-1 from inference-step T to form (part of) the input to layer N at inference-step T. (To have a model learn to do something with this "tool", you just need to ensure its training is measuring predictive error over multi-token sequences generated using this multi-step working-memory persistence.)

...but we aren't currently allowing models to do that. Models currently "have amnesia" between steps. In order to do any kind of asynchronous multi-step thinking, everything they know about "what they're currently thinking about" has to somehow be encoded — compressed — into the observed-space sequence, so that it can be recovered and reverse-engineered into latent context on the next step. And that compression is very lossy.

And this is why ChatGPT isn't automatically a better WolframAlpha. It can tell you how all the "mental algorithms" involved in higher-level maths work — and it can try to follow them itself — but it has nowhere to keep the large amount of "deep" [i.e. latent-space-level] working-memory context required to "carry forward" these multi-step processes between inference steps.

You can get a model (e.g. o1) to limp along by dedicating much of the context to "showing its work" in incredibly-minute detail — essentially trying to force serialization of the most "surprising" output in the latent layers as the predicted token — but this fights against the model's nature, especially as the model still needs to dedicate many of the feed-forward layers to deciding how to encode the chosen "surprising" embedding into the same observed-space vocabulary used to communicate the final output product to the user.

Given even linear context-window-size costs, the cost of this approach to working-memory serialization is superlinear vs achieved intelligence. It's untenable as a long-term strategy.

Obviously, my prediction here is that we'll build models with real inference-framework-level working memory.

---

At that point, if you're adding mutable weights and working memory, why not just admit defeat with Transformer architecture and go back to RNNs?

Predictability, mostly.

The "constant-bounded compute per output token" property of Transformer models, is the key guarantee that has enabled "AI" to be a commercial product right now, rather than a toy in a lab. Any further advancements must preserve that guarantee.

Write-once-per-layer long-term-durable mutable weights preserve that guarantee. Write-once-per-layer retained-between-inference-steps session memory cells preserve that guarantee. But anything with real recurrence, does not preserve that guarantee. Allowing recurrence in a neural network, is like allowing backward-branching jumps in a CPU program: it moves you from the domain of guaranteed-to-halt co-programs to the domain of unbounded Turing-machine software.

lowsong• 2 years ago

Last week I had to caution a junior engineer on my team to only use an LLM for the first pass, and never rely on the output unmoderated.

They're fine as glorified autocomplete, fuzzy search, or other applications where accuracy isn't required. But to rely on them in any situation where accuracy is important is professional negligence.

rglover• 2 years ago

Yes, but not absolutely.

LLMs are a valuable tool for augmenting productivity. Used properly, they do give you a competitive advantage over someone who isn't using them.

The "dead end" is in them being some magical replacement for skilled employees. The levels of delusion pumping out of SV and AI companies desperate to make a buck is unreal. They talk about chat bots like they're already solving humanity's toughest problems (or will be in "just two more weeks"). In reality, they're approximately good at solving certain problems (and they can only ever solve them from the POV of existing human knowledge—they can't create). You still have to hold their hand quite a bit.

This current wave of tech is going to have an identical outcome to the "blockchain all the things" nightmare from a few years back.

Long-term, there's a lot of potential for AI but this is just a significant step forward. We're not "there" yet and won't be for some time.

ein0p• 2 years ago

My take is that even if AI qualitatively stops where it is right now, and only continues to get faster / more memory efficient, it already represents an unprecedented value add to human productivity. Most people just don't see it yet. The reason why that is, is because it "fills in" the weak spots of the human brain - associative memory, attention, working memory constraints, aversion to menial mental work. This does for the brain what industrialization did for the body. All we need to do to realize its potential is emphasize _collaboration_ with AI, rather than _replacement_ by AI, that the pundits currently emphasize as rage (and therefore click) bait.

arvindrajnaidu• 2 years ago

AI cannot use composition?

karaterobot• 2 years ago

Does anyone seriously think that results of any current approaches would suddenly turn into godlike, super-intelligent AGI if only we threw an arbitrary number of GPUs at them? I guess I assumed everyone believed this was a stepping stone at best, but were happy that it turned out to have some utility.

bgnn4• 2 years ago

What I funny is the discussion recolve a lot around software development, where LLMs excel at. Outside this and creating junk text, like a government report, patent application etc they seem to be pretty useless. So most of the society doesn't care about it and it's not as big as a revolution as SWEs think it is at the moment and the discussion for future is actually philosophical: do we think the trend of development continue or we will hit a wall.

garspin• 2 years ago

LLMs are text compression algos. They are very good at memorising & retrieving text related to the user input. They can even pass bar exams based on that - some misinterpret that as intelligence.

However, no amount of scaling will change a text memorisation algo into a symbolic reasoning or composability algo, both of which are necessary for progress towards AGI.

So yes, LLMs are a dead end in the quest for AGI. However, they have their uses as a google/stackoverflow replacement.

gadr90• 2 years ago

I asked ChatGPT to replace "current AI" and synonyms with "HUMANS" and I'm satisfied. My favorite revised sentences:

"Does HUMANS represent a dead end?"

"HUMANS should not be used for serious applications."

"HUMANS are unmanageable, and as a consequence their use in serious contexts is irresponsible."

"HUMANS have no internal structure that relates meaningfully to their functionality."

"HUMANS have input and state spaces too large for exhaustive testing."

"HUMANS do not allow verification by parts (unit testing, integration testing, etc)."

"HUMANS have faults, but even their error behaviour is likely emergent, and certainly hard to predict or eradicate."

"HUMANS have no model of knowledge and no representation of any ‘reasoning.’"

"HUMANS represent a dead end, where exponential increases of training data and effort will give us modest increases in impressive plausibility but no foundational increase in reliability."

"HUMANS cannot be developed, or reused, as components."

"There is no possibility for stepwise development — using either informal or formal methods — for HUMANS."

and my favorite:

"In my mind, all this puts even state-of-the-art HUMANS in a position where professional responsibility dictates the avoidance of them in any serious application."

kukkeliskuu• 2 years ago

Whether it is a dead end or not is a question of definition. If something is not economically self-sustaining by itself, is it a dead end? How about if people keep doing it anyway, for other reasons? What if that includes pretending it is not a dead end?

For example, there is a considerable incentive to flood resources to "current AI". As a consequence, enough people have a vested interest to participate in what might be called a shared illusion in "current AI is not a dead end". If enough people participate in such a shared illusion for 10+ years, and the illusion has real-life consequences, is it really a dead end?

I believe that "stochastic parrot" is actually a compliment about the current AI. The sweet spot for current AI appears to be as little stochasticity in the output as possible. Randomness in the output is inversely proportional to the benefit you can gain from the AI.

Value is current AI is gained by allowing randomness in the input, i.e. the training data and the form of questions you can ask it.

For example, you can create an useful AI support system that regurgitates answers to FAQs back to the people, so they can ask questions about the FAQs instead of browsing them.

Such a tool is useful because humans are energy-optimizing, i.e. lazy, i.e. avoid going through the FAQs themselves, i.e. are unwilling to pay for the answer with their attention, i.e. using their own brain to do pattern recognition.

By avoiding directed pattern recognition, or more specifically avoiding consciously directing your attention as much as possible, people are not practicing this ability. Over time, they become less and less capable of conciously directing their attention.

As a consequence, this ability becomes even more scarce resource, making "current AI" more and more useful.

So it is less about AI getting more intelligent, but people getting less intelligent, and thus AI becoming relatively more intelligent.

niemandhier• 2 years ago

I call this “The bitter cycle”, after the famous essay.

1. Someone finds a new approach, often based on intuition or understanding.

2. People throw more data at it claiming that only data matters.

3. The new method eventually saturates, everybody starts talking about a new ai winter.

We had this with: perceptrons, conv nets, RNNs. Now we see it with transformers.

My guess is the next iteration will be liquid networks or KANs, depending on which one we figure out how to train efficiently first.

The good thing is that people have been working to build an understanding of why these things work for the last 20 years, so the period between the cycles gets shorter.

Maro2• 2 years ago

The perspective of the article seems to be of a person who has not worked in cookie-cutter software engineering environments, which is what 99% of software engineering is. Here formal methods and verification are irrelevant, nobody has heard of it, and nobody cares. It's about churning out some crappy internal webapp or mobile app or data eng/science pipeline at the lowest cost by some soulless bigco (or startup). LLMs are already super useful for this.

Also, the argument about LLMs being black-box is a miss, because LLMs are writing software, they are not the software; programmers writing software are also black boxes. Also, there's nothing in the way of running formal methods on the software produced by an LLM, it will fail the same as it fails on software written by humans, since most formal verification doesn't even make sense for 99% of software.

Also, anybody who has used LLMs as aids for writing simple/smaller chunks of code (and other documents) knows that they're super useful (sometimes magic). It's like Steve Ballmer saying the iphone is a joke in 2007.

EerkeBoiten10• 2 years ago

Author here. Fair enough on my industry experience. But I hope components, unit testing, regression testing, etc aren't as easily dismissed in real SE environments - no trouble believing formal methods and verification are off the radar. The article is not about using AI to write code (which may work to some level of satisfaction for some people) but about using AI as code.

Maro2• 2 years ago

Thanks for your reply! Hope my comment wasn't offensive.

I definitely think components, unit testing, regression testing are good things and are done at good software houses. In my experience however, most of these things are mostly cargo culted at best in a many other environments.

When I wrote my comment I was wondering about the "AI to write code" vs "AI as code" point. In my vocabulary, "AI as code" would be "Data Science models", like a ranking engine for ads in a newsfeed? I certainly understand the idea of having an AI "emulate" an application like Word.exe or Doom.exe, and there's been research into this direction, but as far as I can tell that is not the general direction the industry is headed in --- rather it's the "AI to write code" direction.

HeavyStorm• 2 years ago

No.

Does current AI represent a dead end?

Related Stories