Ask HN: Is anybody getting value from AI Agents? How so?

https://news.ycombinator.com/item?id=39886178

by reilly3000 • 2 years ago

170 149 2 years ago

news.ycombinator.com

I saw a lot of initial buzz about the promise of agent based workflows and it seemed to be the obvious way to get LLMs to the edge of decision making and leverage many specialized models. It seems the chatter has died down but there are growing projects out there in the space. Before I invest the time to explore and work with tools I’d love some feedback from the community and if and how they are being used.

mattew• 2 years ago

I’ve found the OpenAI assistants API not really up to snuff in terms of predictable behavior yet.

That said, I’m very bullish on agents overall though and expect that once they get their assistants behaving a bit more predictably we will see some cool stuff.

It’s really quite magical to see one of these think through how to solve a problem, use custom tools that you implement to help solve it, and come to a solution.

manojlds• 2 years ago

Eerily quiet here.

Scratcher368653• 2 years ago

It is, isn't it?

spxneo10• 2 years ago

I'm also noting this but could be because its Sunday?

or was GP implying lot of people are gettin funded to build agents in a thread where the consensus is they dont work well enough for people to pay for

zogrodea• 2 years ago

Not sure if this is sarcasm, but the thread was only posted 20 minutes ago and has 9 replies already. I personally am tired of AI/LLM news but it still seems popular from this thread.

hhh• 2 years ago

I haven’t found any useful agent workflows, and I’ve not found a tool that’s more productive for me (doing arch/design/implementation of systems) than just copy and pasting from the Playground.

JCM9• 2 years ago

I’ve seen a lot of attempts but nothing that worked really well. Using an agent as a glorified search engine can work, but trying to replace actual humans to handle anything but the most standard use cases is still incredibly hard. There’s a lot of overhyped rhetoric at the moment around this tech, and looks like we’re heading into another period of post-hype disillusionment.

Legal angles here also also super interesting. There’s a growing body of scenarios where companies are held accountable for the goofs of their AI “assistants.” Thus we’re likely heading for some comical train wrecks as companies that don’t properly vet this stuff set themselves up for some expensive disasters (eg think the AI assistant doing things that will get the company into trouble).

I’m bullish on the tech, but bearish on the ability of folks to deploy it at scale without making a big expensive mess.

spxneo10• 2 years ago

Extrapolating on your legal comment, copyright is the current issue.

Right now the USCO says if you use a large model that used copyrighted material to train it, you cannot copyright the generated art. This applies to not just art but might be for code as well. So I wonder what the legal liabilities are say for a publicly traded company to use Midjourney to generate content that is also copyrighted.

ex) it is possible to generate movie characters that we instantly recognize if you use non-english words which pretty much nail in the coffin for safe harbour status granted under DMCA

dragonwriter• 2 years ago

Right now the USCO says if you use a large model that used copyrighted material to train it, you cannot copyright the generated art.

Unless the Copyright Office has come out with a newer rule, it says if you use a model that does the usual creative parts of making the output you can’t copyright the output, because it is not a work of human authorship. (And that if a work is mixed AI/human, the AI use must be disclosed in copyright registration to avoid copyright protection being applied to elements that are not human work.) “Large model” and “copyrighted material used to train it” are not the issues.

stillTwerkin• 2 years ago

Eh, who cares? I get money from my job and can save money entertaining myself with generative art. Even if I can’t copyright it, fine, because it’s an outdated political and economic cudgel used to expropriate and exploit more often than not

The past is so often proven wrong and was so much less civil, we can never actually ask James Madison if we did right by his notion the future owes the past. We can only decide for ourselves we align with the long dead’s philosophy.

Where is the value in so carefully conserving in such detail the past? I’m not saying we end history as a field of study but as an obligation of daily life.

The most important lesson of history, imo, is prevent nation states or whatever might take their place from having the ability to wage wars. Most of their philosophy is indecipherable given our life experience is vastly different than theirs. Would they see this world and come to the same conclusion they did?

jpollock• 2 years ago

The GP might not be exclusively talking about copyright. They might also be talking about how Air Canada lost a lawsuit about whether or not they needed to honor their chatbot's hallucinations.

https://www.theguardian.com/world/2024/feb/16/air-canada-cha...

echelon3• 2 years ago

AI agents suck. It's too early.

The first dominoes to fall will be art, music, and film. These don't require perfection and can be iterated on by the creator. The tools become a method in the process.

Everyone in the agent space is bumping around until the next big innovation that actually unlocks the technique sparks a race to productize. Maybe some will get lucky and have a fast pivot. Or maybe they'll run out of funds before the big breakthrough.

spxneo10• 2 years ago

right now the expected future cashflow from whoever "wins" is infinite justifying astronomical amount of capital expenditure. ex) Microsoft's $100 billion supercomputer

Sometimes I get they are already sitting on some ground breaking stuff and slowly releasing it to test our responses and get feedbacks.

I'm certain they will be reading this thread but if one theme repeats itself is the lack of trust in the output of the agents and the companies creating it.

nonrandomstring2• 2 years ago

AI agents suck.

The idea does not suck. On paper it's great. Just tell an "agent" what you want and off it goes to get it. And it's possible. LLMs open the interface to search planning and selection algorithms that have been around for 40 years and are mature. You could have this tomorrow.

The assumption is that people want it.

Tech business has come a long way exploiting people's vices, specifically laziness of thought we call "convenience". But at heart, tech is still seen as a tool, to empower people, to give them agency.

Agents subtract agency.

It's too early.... the big breakthrough.

Hoping for progress against human psychology seems a fool's errand.

ganzuul2• 2 years ago

Talented artists see their creative output drown under a torrent of commercial and mass-produced art. There are dominoes to fell but we should not be reckless about it.

With postmodernism art deconstructed its duty and without duty society does not grant rights. It is a big problem because art is often visionary about the future. We don’t know that we even have a future without vision.

roncesvalles• 2 years ago

A major issue with generated art is precision - you never get perfectly repeatable details.

Let's say if you try to make a Family Guy episode using generated video, the characters would come out slightly different in every scene. You could generate 100 outputs for every scene and try to pick the best one but it still won't be very good.

wslh• 2 years ago

We are doing that internally, so I think it is now more a craft than a "product". For example, we look at lot of specific codebase repositories (e.g. GitHub) and try LLMs over the diff just before and after a security code audit was done.

Another one is listening to many social media (e.g. Twitter) posts to sense if there is a business opportunity. SDRs scan the results in an Slack channel manually but based on these signals.

Finally, this is now a workflow but we did this [1] that is a piece in our work.

[1] https://news.ycombinator.com/item?id=39280358

tracer4201• 2 years ago

At my work, I have colleagues who speak English as a second language. Many of them are using LLMs to up their document and other writing.

It’s actually quite awful. It’s obvious the text is LLM generated because of the verbose, generic writing style. It communicates clearly but without substance. Not gonna lie, I secretly judge these people.

trzy• 2 years ago

Aren’t agents bottlenecked by the underlying models? I’ve read that the number of “chain of thought” steps needed is proportional to task complexity. And if each step has the same probability p of success, probability of success is p^n, where n is the number of steps needed (potentially high). At a 99% success rate per step and 5 steps that’s a 95% overall success rate. 90% drops down to 60%. Not sure what the real numbers are but this seems like it could be a problem without significantly more intelligent ML models?

Filligree• 2 years ago

Error-checking and recovery is a potential solution here. Not a well understood one, and might still need higher intelligence than we've got, but-

If your math worked out, then humans couldn't work either.

pphysch2• 2 years ago

If you have a "super-agent" AI that is capable of recovering a business process from an error state, why not just use that agent in the first place?

babelfish3• 2 years ago

Are there any papers on this? I’d be interested in reading more.

JohnKemeny2• 2 years ago

There's a paper by Papadimitriou (from Logicomix fame) and some collaborators that the transformer model is incapable of solving certain simple problems, and if done by Cost, it needs exponentially many steps.

The paper is currently only available at Arxiv (ie not yet peer-reviewed), but given that it is Papadimitriou, I would be inclined to believe the results.

https://arxiv.org/abs/2402.08164

babelfish3• 2 years ago

Klarna used the OpenAI Assistants API to automate the work of ~7000 support agents.

nicce• 2 years ago

I believe you have one zero too much over there

https://www.klarna.com/international/press/klarna-ai-assista...

babelfish3• 2 years ago

Oops, thanks!

kerkeslager• 2 years ago

...which is not necessarily evidence that Klarna has obtained value from OpenAI Assistants API. It's quite possible (likely, even) that this has effectively been a complete removal of support.

I haven't used Klarna, but a few other products I've used are unsupported now because human support were replaced with AI "support" that is completely useless.

kennethologist• 2 years ago

Apparently Pieter Levels: " Interior AI now has >99% profit margins

- GPU bill is $200/month for 21,000 designs per month or about 1¢ per render (no character training like Photo AI helps costs) - Hosted on a shared VPS with my other sites @ $500/mo, but % wise Interior AI is ~$50 of that

+= $250/month in costs

It makes about $45,000 in MRR and so $44,730 is pure profits! It is 100% ran by AI robots, no people

I lead the robots and do product dev but only when necessary"

https://twitter.com/levelsio/status/1773443837320380759

ShamelessC4• 2 years ago

Is that an agent based setup? Seems like it’s using a few different models wired together manually.

iAkashPaul2• 2 years ago

His robots are just automated scripts which do the bulk repetitive tasks unlike the agents that comment OP thought.

kylecazar• 2 years ago

And yet "Unfortunately, we cannot offer refunds as costs incurred for generating AI images are extremely high."

dvfjsdhgfv• 2 years ago

Yeah, he claims because at any point in time there are people redesigning their interiors. I'd say that at any point in time there are people you can convince to give you their money, and if you don't offer refunds, it's not far from a scam.

naijaboiler• 2 years ago

exactly 99% gross profit, but can't offer refunds due to cost

conjectures• 2 years ago

Gent is shilling his book about passive income or whatever. Sure I believe his numbers.

balls187• 2 years ago

$45,000 in MRR and so $44,730

I’ve found a lot of these numbers from people selling passive income methods are extrapolated.

jokethrowaway2• 2 years ago

Levels has been one of the most open entrepreneurs out there. I'd be surprised if he lied on revenue

All of that just to sell a book?

paxys• 2 years ago

This guy makes money by selling "how to get rich using AI" courses and marketing himself on social media (which he is phenomenal at). I'm not really inclined to believe his sales numbers.

Mystery-Machine• 2 years ago

He is NOT selling any courses. Can you please point me to any of his courses? He has a book he wrote about making software/projects called Make. This book is several years old and doesn't mention AI.

GaryNumanVevo• 2 years ago

I roll my eyes every time I see a tweet of his. "Entrepreneur influencers" are uniquely unbearable.

suchintan2• 2 years ago

I don't think ai agents are good enough to replace every job today, but they're starting to nip at the more junior / menial knowledge jobs

I've seen a lot of success come from AI sales agents, just doing basic SDR style work

We're having some success automating manual workflows for companies at Skyvern, but we've only begun to scratch the surface.

I suspect that this will play out a lot like the iPhone era -- first few years will be a lot of discovery and iteration, then things will kick into superdrive and you'll see major shifts in user behavior

JohnKemeny2• 2 years ago

I don't think ai agents are good enough to replace every job today

You mean any.

You can fire a human knowledge worker for not doing their job correctly, but what are you gonna do when you only have LLMs and realize they can't do their job correctly?

suchintan2• 2 years ago

I think there are some they're good enough at today. Auto generating meeting notes + AI context, auto responding / following up to emails, filling out forms (we do that pretty well at Skyvern with high accuracy)

lpapez• 2 years ago

Fire 8 out of 10 of your knowledge workers and have the other remaining 2 review and fix LLM output.

Basically the same what we have now, except the grunt workers will be replaced by the machine.

andy992• 2 years ago

junior / menial knowledge jobs

Not junior, unthinking. It's like outsourcing/offshoring. You're getting the equivalent of someone who doesn't care about what they're doing, not somebody inexperienced. I'm not saying it doesn't have a place, just commenting on the framing.

salomon812• 2 years ago

I've been a bit disappointed by the AI. I'll admit going in with low expectations (I know about the whole AI summer/winter cycle) and I was blown away that ChatGPT could play Jeopardy! with just a prompt since I remember being blown away by Watson and AlphaGo. But then I had it help me write a letter, and by the time I got it to do anything useful, I basically had to write an outline for it, and then I realized I had already done the hard part. I asked it to write some boilerplate code for an interface to the Slack API in Python, but it used a deprecated API, and it assumed I had a valid token. Turns out Slack has lots of different kinds of tokens and I was using the wrong one, and the AI couldn't help me figure that out. After that, I remembered the story about pain point for radiologists. They don't need help diagnosing cancer, they need help with their internet connectivity.

psalmadek• 2 years ago

I’m here to read the comments but I think it is too early to give a statement of fact.

AnimalMuppet• 2 years ago

In itself, that is an interesting statement of fact: Whatever value AI agents are going to deliver, they aren't delivering yet. The jury is still out.

kwinkunks• 2 years ago

I've been disappointed by my few experiments with Langchain's agent tooling. Things I have experienced:

- The pythonrepl or llm-math agent not being used when it should be and the agent returning a wrong or approximate answer.

- The wikipedia and webbrowsed agents doing spurious research in an attempt to answer a question I did not ask (hallucinating a question, essentially).

- Agents getting stuck in a loop of asking the same question over and over until they time out.

- The model not believing an answer it gets from an agent (eg using a Python function to get today's date and not believing the answer because "The date is in the future").

When you layer all this on top of the usual challenges of writing prompts (plus, with Python function, writing the docstring so the agent knows when to call it), wrong answers, hallucination, etc, etc, I'm unconvinced. But maybe I'm doing it wrong!

ruined• 2 years ago

copilot has been useful to me for the boring parts of writing tests, but it needs a lot of review. other than that, dogshit

numberless• 2 years ago

I personally use an AI FAQ bot to automate FAQ questions in some of my Discord servers. It doesn't always work as well as a human answering but it does help, in most cases. In other words, AI Agents can be very helpful but can only be trusted/useful to a limited extent compared to humans.

airstrike• 2 years ago

I get a lot of value out of Copilot and GPT4 for coding, but that's about it.

It's true that have to wrestle a lot with them to get them to do what I want for more complex tasks... so they are great for certain tasks and terrible for others, but when I'm in Xcode, I dearly miss vscode because of Copilot autocomplete, which I guess is an indication that it adds some value

One unexpected synergy has been how good GPT4 is at explaining why my rust code is so bad, thanks to the very verbose compiler messages and availability of high quality training data (i.e. the great rust code in the wild)—despite GPT4 not always being great at writing new rust code from a blank file.

Part of me thinks in the future this loop is going to be a bit more automated, with an LLM in the mix... similar to how LSPs are "obvious" and ubiquitous these days

On an unrelated note, I also wrote a small python script for translating my Xcode project's localizable strings into ~10 different languages with some carefully constructed instructions and error checking (basically some simple JSON validation before OpenAI offered JSON as a response type). I only speak ~2 of the target languages, and only 1 natively, but from a quick review the translations seemed mostly fine. Definitely a solid starting point

ShamelessC4• 2 years ago

That’s not an agent based setup.

paradite• 2 years ago

Agents still very new and nothing that works for production yet.

Specifically on using AI for coding, I wrote about different levels of AI coding from L1 to L5, we are still at L2/L3 stage for mature and production ready tech. Agents are L4/L5:

https://prompt.16x.engineer/blog/ai-coding-l1-l5

wkirby• 2 years ago

The best quote I’ve heard from our clients is “don’t trust AI with anything you wouldn’t trust a high schooler to do.”

That line of reasoning has held true across basically every project we’ve touched that tried to incorporate LLMs into a core workflow.

drewcoo• 2 years ago

> “don’t trust AI with anything you wouldn’t trust a high schooler to do.”

Then they should be great for making fast food, staffing amusement parks, and seasonal farm labor.

They don't seem to be good for those things either.

[Edit to add: the value high schoolers bring to jobs through non-cognitive abilities, which AIs lack.]

clevergadget• 2 years ago

to be fair neither are high schoolers

dudus• 2 years ago

Opinions mine based on learning from scratch this space in the last couple months only.

I feel like these architectures built on top of last gen LLMs are mostly useless now.

The current gen jump was significant enough that creating a complex chain of thought with RAG on last gen usually is surpassed by 0 shots on current gen.

So instead of spending time and money building it it's better to focus on 0-shot and update your models to the latest version.

Feeding LLM outputs into other LLM inputs IMHO just will increase the bias. Initially I expected to mix and match different models to avoid it but that didn't work as much as I expected.

It depends a lot on your application honestly.

spxneo10• 2 years ago

but aren't current gen 0 shots gated and throttled? or has things changed for azure openai

stavros3• 2 years ago

I just (as in, five minutes ago) hooked GPT 4 up to my 3D printer and it's fantastic, I use an ESP32 Box and I can ask it what files I have on my printer, I can ask it to print a file, I even added calendar integration so it can read me my events and add new ones. I love it.

All that's left is for someone to bundle it all up into a nice package, and we'll be in the future.

mechagodzilla2• 2 years ago

I use a 3D printer all the time, but I almost never print a file more than once. It also takes a couple of seconds to select a file with the built-in interface, and then it usually takes anywhere from 30 minutes to a day or more to print. What's your usage model that putting a GPT4 instance between you and the printer is somehow helping things? This feels like someone saying "Now I can e-mail my kleenex box to see how much kleenex it has."

stavros3• 2 years ago

I had some small prints that took ten minutes each, that I needed a lot of, so I figured it would be nice to talk to my ESP box and have it launch prints.

tomrod• 2 years ago

Some. We are building some new processes from the ground up and will use Agents as a first draft contributor. This is typically where we find the most slowdown. And we will consistently search for the word "delve" as a misspelling :)

tuckerconnelly3• 2 years ago

I built an AI-agents tech demo[1], and am now pivoting. A few thoughts:

* I was able to make a simple AI agent that could control my Spotify account, and make playlists based on its world knowledge (rather than Spotify recommendation algos), which was really cool. I used it pretty frequently to guide Spotify into my music tastes, and would say I got value out of it.

* GPT-4 worked quite well actually, GPT-3.5 worked maybe 80% of the time. Mixtral did not work at all, aside from needing hacks/workarounds to get function-calling working in the first place.

* It was very slow and VERY expensive. Needing CoT was a limitation. Could easily rack up $30/day just testing it.

My overall takeaway: it's too early: too expensive, too slow, too unreliable. Unless you somehow have a breakthrough with a custom model.

From the marketing side, people just don't "get it." I've since niched down, and it's very, very promising from a business perspective.

[1] https://konos.ai

spxneo10• 2 years ago

what makes it slow? is it because they throttle your api key?

itake2• 2 years ago

Our p99 for gpt4 is 3s. Images take up to 50s.

tuckerconnelly3• 2 years ago

Chain of thought takes time to generate all the characters. If you do a chain-of-thought for every action and every misstep (and you need to for quality + reliability), it adds up.

andy992• 2 years ago

I think it's destined to fail because it basically moved AI back into the "rules based" realm. Deep learning is a decent cognitive interface - like making a guess at some structure out of non-structure. That's where the magic happens. But when you take that and start using rules to chain it together, you're basically back to the same idea as parsing semi-structured data with regex and/or if statements. You can get it to work a bit but edge cases keep coming along that kill you, and your rules will never keep up. For simple cognitive tasks, deep learning figures out enough of the edge cases to work pretty well, but that's gone once you start making rules for how to combine predictions.

thekumar• 2 years ago

I totally agree with this. I have been arguing with folks that current Reactflow based agent workflow tools are destined to fail, and more importantly, missing the point. Stop forcing AI into structured work.

I do think AI "agents" (or blocks as I like to think of them) unlock the potential for solving unstructured but well-scoped tasks. But it is a block of unstructured work that is very unique to a problem, and you are very likely to not find another problem where that block fits. So, trying to productize these AI blocks as re-usable agents is not that great of a value prop. And building a node based workflow tool is even less of a value prop.

However, if you can flip it inside out and build an AI agent that takes a question and outputs a node based workflow. But the blocks in the workflow are structured pre-defined blocks with deterministic inputs and outputs, or a custom AI block that you yourself built, then that is something I can find value in. This is almost like the function calling capabilities of GPT.

Building these block reminds me of the early days of cloud computing. Back then the patterns for high availability were not well-established and people that were sold on the scalability aspects of cloud computing and got onboard without accounting for service failure/availability scenarios and the ephemeral nature of EC2 instances were left burned, complaining about the unfeasibility of cloud computing.

1oooqooq• 2 years ago

so, no?

tuckerconnelly3• 2 years ago

There's value, but it's too expensive, too slow, and too unreliable right now to be feasible from a business perspective.

hubraumhugo• 2 years ago

The term "AI agents" might be a bit overhyped. We're using AI agents for the orchestration of our fully automated web scrapers. But instead of trying to have one large general purpose agent that is hard to control and test, we use many smaller agents that basically just pick the right strategy for a specific sub-task in our workflows. In our case, an agent is a medium-sized LLM prompt that has a) context and b) a set of functions available to call. For example we use it for:

- Navigation: Detect navigation elements and handle actions like pagination or infinite scroll automatically.

- Network Analysis: Identify desired data within network calls.

- Data transformation: Clean and map the data into the desired format. Finetuned small and performant LLMs are great at this task with a high reliability.

The main challenge:

We quickly realized that doing this for a few data sources with low complexity is one thing, doing it for thousands of websites in a reliable, scalable, and cost-efficient way is a whole different beast.

The integration of tightly constrained agents with traditional engineering methods effectively solved this issue for us.

erichi• 2 years ago

Why using llm to chose a proxy if you can just rotate starting from the cheapest based on not getting 403?

bsenftner2• 2 years ago

I've found that while agents cannot replace anyone, they can sure help with the use of various things.

First, we know these AIs are trained with data from the general Internet, and that data is vast.

Second, the general Internet contains owner manuals and support forums for practically every active product there is, globally. These are every possible product too: physical products, virtual products like software or music, and experience products like travel or education. Between the owner’s manuals and the support forums for these products there is extremely deep knowledge about the purpose, use and troubleshooting of these products.

Third, one cannot just ask an LLM direct deep questions about some random product and expect a deep knowledge answer. One has to first create the context within the LLM that activates the area(s) of deep knowledge you want your answers to arise. This requires the use of long form prompts that create the expert you want, and once that expert is active in the LLM’s context, then you ask it questions and receive the deep knowledge answers desired.

Fourth, one can create an LLM agent that helps a person create the LLM agent they want, the LLM agent can help generate new agents, and dependency chains between different agents are not difficult at all, including information exchange between groups of agents collaborating on shared replies to requests.

And last, all that deep information about using pretty much every software there is can be tapped with careful prompting to create the context of an expert user of that software, and experts such as these can become plugins and drivers for that software. It's at our finger tips...!

nottorp• 2 years ago

What's an agent based workflow? :)

I use LLMs as a glorified search engine. That was better than web search at some point, I'm not sure the publicly available LLMs are that good any more. Gemini seems to be extremely worried to not offend anyone instead of giving me results lately.

At least it's still useful for 'give me the template code for starting an XXX' ...

morkalork• 2 years ago

Is it something like this?

You are a <blank assistant>, the user has requested <input>. Here are a list of possible actions and their descriptions. Choose the most appropriate action for the user's request. Parse the following from the request...

In a loop, and you execute whichever action is selected after?

jaxomlotus2• 2 years ago

We built a conversion rate optimizing AI Agent and saw about 45% click through rate lift on our own homepage. In other beta testing companies that used it, we saw a similar average and range has been +15%-175%. Agent (AB3.ai) can be tried here: https://AB3.ai

spxneo10• 2 years ago

you know these driveby comments writing a bit of contextually relevant bit that always ends with a me-too link are getting tiring. To the point where I see it here, I begin to suspect things aren't going well in your other sales/marketing funnels.