Generative AI is good at cooperating with people and bad at full automation

https://skybrian.substack.com/p/generative-ai-is-good-at-cooperating

by skybrian • 3 years ago

98 58 3 years ago

skybrian.substack.com

faizshah2• 3 years ago

I agree with the overall idea, but we don’t need to limit the AI to human collaborators.

I think the future lies in the Copilot approach of giving the Agent relevant context by building smarter prompt generation pipelines. The AI are really good when given enough context and they are bad at recalling information from their own training corpus. When you combine multiple AI agents together with good context and tools like in the LangChain approach these AI agents suddenly become really good at full automation.

Which imo is the path to AGI.

HanClinto• 3 years ago

Do you have examples of combining multiple AI agents together with context and tools? I'd like to learn more about what you're referring to.

faizshah2• 3 years ago

This is a basic example of a Reason and Act (React) agent: https://python.langchain.com/en/latest/modules/agents/agents...

Search is a tool that provides more context to the agent in order to answer a human task. By combining many agents together we can solve complex tasks.

TMWNN• 3 years ago

Cooperation with people is what I thought of when I heard of the forthcoming Business Chat feature for Microsoft Teams <https://news.ycombinator.com/item?id=35182827>. From what I have read, it is going to produce summaries of conversations. That's not quite what I'd imagined, which is something like this:

----

A: ... and that is why I think we should go with option 1.

B: No, the points you mentioned support my case for option 2.

C: Nothing you guys have said changes my mind about option 3 being best.

D: Business Chat, what do you think?

BC: Based on this discussion, and my research, option 1 seems more realistic but option 2 would be more profitable if possible. My reasons are ...

C: Business Chat and you guys all don't understand point N, which is the main reason why option 3 is best.

B: Higher profit is exactly why I think option 2 is the way to go.

A: No, our rival is going to hit the market next month. We need to get something out there ASAP. Option 1 can do this.

D: You've all given me things to think about. Thank you for coming. Business Chat, email me a summary of the meeting, and set up a followup meeting for Tuesday 3pm.

----

That is, AI used as a colleague/assistant, not necessarily subordinate but not seen as omniscient, either; another viewpoint to consider. When will the above be feasible? This year? Five years?

babyshake• 3 years ago

This type of thing really is science fiction, in the sense of Data or C-3PO sharing their (often ignored) opinion or assessment.

layer8• 3 years ago

People would probably tend to grow an aversion to Business Chat for its smartypants opinions. Microsoft may want to avoid that. ;)

NovaDudely2• 3 years ago

In a way you could use these AI systems similar to kintsugi. Or in the same way a good wood sculptor uses the flow of the wood rather than working against it.

Using AI to provide a basic frame work on which folks can iterate, manipulate or redact is not such a dystopian view of these things.

ChatGTP2• 3 years ago

Which would be a dream if it stops there, but it sounds like we're fairly hell bent on taking it further as fast as possible?

bloppe• 3 years ago

Fersher

NovaDudely2• 3 years ago

Absolutely. If reading any of the computer technology pessimists of the last 50 years have taught me anything is that they are usually right to some degree.

dclowd99012• 3 years ago

I don’t understand what we have to gain from a system that generates code that needs human verification. It’s like Tesla’s autopilot all over again. Either it can drive itself or it can’t — if it can’t absolute, 100% reliably manage itself, why are we even pursuing it? I bet a more clever person than I could actually mathematically prove that neural network based models could never provide 100% reliable results.

examplary_cable• 3 years ago

What about AI collaborating with AI?

agnosticmantis• 3 years ago

It used to be that classical software was useful for verification, but not so good at generating ideas.

With LLMs we have the opposite: machines are now good at idea generation and exploration, but their ideas need to be verified by a human (or classical logic-based software).

jeron• 3 years ago

So all we need to do is have LLMs generate classical logic-based software to qualify its own ideas

TaylorAlexander• 3 years ago

I’m quite confident they’re using GitHub copilot or similar inside of OpenAI while working on improving GPT-4. So yeah, that’s happening.

chaxor• 3 years ago

There are plenty of different ways this is happening. 1) already happened with Galactica - the model included python scripts as part of it's "working memory" as part of a <work> token, which could be used in a container 2) via copilot, and copilot 365, as someone stated elsewhwre 3) GPT-> Wolfram alpha as a plugin, which will be out very soon

Galactica is the only one that is already (technically) available

dclowd99012• 3 years ago

But, by virtue of how they work, are they any good at generating novel ideas?

eximius2• 3 years ago

That raises an interesting point. On the one hand, novelty is in the eye of the beholder, if you will. What has been done before might still be new to many. On the other, randomness is built in and language has a remarkable capacity for uniqueness (see statistics about novel queries on Google). So it's possible and likely for LLMs to generate novel sentences. Whether those are valuable is a harder thing to think about.

esjeon2• 3 years ago

Not really, but it still helps you a lot with breaking your own bias.

dougSF702• 3 years ago

Since the LLMs are probabalistic is it possible for the inference engines to highlight lower probability content. That might at least make them more "human" rather than being confident and inaccurate.

nicognaw2• 3 years ago

Since LLMs generate words one by one, you can only determine whether the next word has a high probability. Some ideas occurred to me earlier, like applying prompt engineering to evaluate if the model is confident about the contents to be generated.

Like "Please reply T if you are confident about the text that you are going to generate, and F if you are not".

But if this could be done by architecting the neural networks, they would perform much better.

pmoriarty4• 3 years ago

"Since LLMs generate words one by one, you can only determine whether the next word has a high probability"

We don't fully understand how they work or what their limits are.

skybrian5• 3 years ago

My guess is that a low-probability token corresponds as much with creativity (it's saying something in an unusual way) as truth, since it's at the word level. Words aren't normally true or false, sentences are. Do LLM's represent their confidence in what they're saying anywhere? I don't think anyone knows. Another mystery.

But another problem is that confidence is a character attribute, not a writer attribute. If you ask an LLM to imitate Richard Feynman it's going to write pretty confidently about physics, despite not knowing as much as him about physics.

This is the equivalent of giving your RPG character high intelligence on the character sheet. Doesn't make you smart!

To make an LLM express confidence consistent with its actual knowledge, it would need to have good self-knowledge and actually use it. So far this doesn't happen automatically. Instead, OpenAI uses reinforcement learning based on what the people at OpenAI think the LLM can do. So that's why it sometimes refuses to answer for some kinds of questions.

That has the most effect on the default character, the "helpful AI assistant". Any other characters you ask for will likely have poorer self-knowledge.

I've read that, mysteriously, more training does make bigger LLM's better calibrated, but the reinforcement learning makes it worse again.

dougSF702• 3 years ago

One other thought I had, was did anyone fact-check the training corpus. It is probable it contained factual errors, which would mean that the trained model would propagate errors some how.

nicognaw2• 3 years ago

Prompt engineering, in my opinion, could be considered "semantic programming."

keyle• 3 years ago

That's a great thought. Although an overarching generalisation.

chx3• 3 years ago

Generative AI is good at ~~cooperating~~ with people

fooling

FTFY

skybrian5• 3 years ago

It follows the "yes, and" rule of improv. This is cooperative if you understand what it's doing. If you want to believe it has any fixed motivations other than that, you fool yourself.

chx3• 3 years ago

I fully understand what it's doing: it exploits the human brain being wired to see patterns where there is none. It vomits up words in such patterns humans have a high tendency to trust them. Of course , there is zero actual information content in it. It's a zeroday on human cognition, a terrible, a terrible danger to society but there's no stopping it. https://futureoflife.org/open-letter/pause-giant-ai-experime...

midjji• 3 years ago

They are good at telling people what they want to hear. There are lots of people like that too, but yeah they generally dont achieve much on their own.

esjeon2• 3 years ago

Right. If you don’t dig deeper, it will give you only generic answers, and even fully incorporate the intention of your questions, strengthening your biases.

palad1n• 3 years ago

Often I can get it to fact-check itself by asking for something in two different ways.

yawnxyz• 3 years ago

I was trying to get ChatGPT to generate a table for all the products I'm trying to put together for Stripe for this conference I'm helping out with, and with all the permutations there's 24 products at different prices.

I fed it the options for all the columns and it couldn't do it! So in the end I resorted to telling it to give me a JS function to generate a CSV, and it did it perfectly. I think I'll end up telling it to give me code that gets me the thing I want (when it comes to data), rather than get the formatted data directly. It sometimes makes weird mistakes, and it's less reproducible.

nomel• 3 years ago

ChatGPT 3.5 or 4?

yohannesk• 3 years ago

That way you also get the added benefit of not sharing your data

obsoletedbygpt• 3 years ago

I'm amazed at how much nay-saying there is on HN about LLMs. This community has bought into so many fads, like crypto, yet when presented with what's clearly going to be the most disruptive technology of your life, you act like it's no big deal. Is this just nervous denial? Are you suddenly incapable of linear (not exponential) extrapolation? Imagine what ChatGPT, Midjourney, or whatever Apple and Google are cooking up will be capable of if they only improve linearly, at say 1% per year?

People ought to be preparing themselves for this as if it were a COVID-like black swan event, except much, much worse. And do not bank on Sam Altman and his fellow tech bro billionaires holding the reins of these job-annihilating machines delivering with their promise of UBI.

weatherlite2• 3 years ago

I get why you're being down voted but your overall point is valid imo. We didn't get any good answers about what happens if this thing keeps getting better.

"People will find new jobs/things to do" doesn't really cut it. It's not really the tech titans fault, the governments really need to start stepping in.

skybrian5• 3 years ago

There are AI boosters making exaggerated claims in their enthusiasm and therefore people take the other side in reaction to them because they’re wrong. The skeptics then sometimes write blanket dismissals in opposition, and that’s wrong too, and you get a loop. That’s just how it goes on the Internet sometimes.

I totally get the motivation and it’s why I started blogging again. To get more interesting conversations, I think we need curiosity while keeping in mind how little we know and how bad we are at predicting the future.

weatherlite2• 3 years ago

I've just read these models are getting to the point they run out of training data. They pretty much scanned most of the internet already. I wonder if we will reach a disappointing situation in 3-4 years where we're at "peak A.I" abilities. Well disappointing for some, for others it means they will get to keep having a job with income :)

inductive_magic• 3 years ago

The current problem isn’t a lack of data. Meaningful data can be augmented with GPTs, just iterate over sparsely represented subjects and enrich them with context-aware samples …generated with GPT.

We are actively leveraging GPT4s brainstorming-ability to generate datasets used for finetuning downstream. The fine tuned models downstream can then be used to augment new training data for more complex root language models, just formalize your quality assurance with common sense DSLs. We’ve reached an upward spiral with quite the exponential feel to it, where it will lead, nobody knows.

throwaway1851• 3 years ago

That’s only going to get you so far. Sparsely represented subjects have have an actual reality you’re trying to model. So while yes, you can generate synthetic data to interpolate between the points that have already been sampled, whether those interpolated points have anything to do with the underlying reality is a matter of chance. If you really try to drill down the existing tools into a specialized problem area, it becomes very clear that they lack a sufficiently informed model of the subject to be useful. That’s why I’ve found that ChatGPT is great at the beginning of a project and increasingly irrelevant as you approach the core technical challenges of the project.

pcwein• 3 years ago

Agree - we're taking a hybrid approach - meaning the gen AI is 'prebuilding' the automation and then the user reviews and possibly corrects it before fully executing - check it out here: bardeen.ai/ai

Generative AI is good at cooperating with people and bad at full automation

Related Stories