I trusted an LLM, now I'm on day 4 of an afternoon project

https://nemo.foo/blog/day-4-of-an-afternoon-project

by nemofoo • 1 year ago

278 199 1 year ago

BigParm• 1 year ago

LLM == WGCM = Wild Goose Chase Model

I ask this question without a hint of tone or sarcasm. You said: "*it’s a junior dev faking competence. Trust it at your own risk.*" My question is simply: "wouldn't you expect to personally be able to tell that a human junior dev was faking competence?" Why should it be different with the LLM?

latexr2• 1 year ago

Obviously, it depends on context. When talking to someone live you can pick up on subtle hints such as tone of voice, or where they look, or how they gesticulate, or a myriad other signals which give you a hint to their knowledge gaps. If you're communicating via text, the signals change. Furthermore, as you interact with people more often you understand them better and refine your understanding of them. LLMs always forget and “reset” and are in flux. They aren’t as consistent. Plus, they don’t grow with you and pick up on your signals and wants.

It’s incredibly worrying that it needs to be explained again and again that LLMs are different from people, do not behave like people, and should not be compared to people or interacted like people, because they are not people.

appleorchard46• 1 year ago

Interestingly your description of social cues you expect to pick up on are the exact sort of social cues I struggle with. If someone says something, generally speaking I expect it to be true unless there is an issue with it that suggests otherwise.

I suppose the wide range of negative and positive experiences people seem to have working with LLMs is related to the wide range of expectations people have for their interactions in general.

layer8• 1 year ago

Not instantly. You’d give the human junior dev the benefit of the doubt at first. But when it becomes clear that the junior dev is faking competence all the time (that might take longer than the four days in TFA — yes I know it’s not exactly comparable, just saying) and won’t stop with that and start being honest instead, you’d eventually let them go, because that’s no way to work with someone.

zitterbewegung2• 1 year ago

I have used LLMs as a tool and I start to "give up" working with it after a few tries. It excels at simple tasks, boilerplate, or scripts but larger programs you really have to know what exactly you want to do.

I do see the LLMs ingesting more and more documentation and content and they are improving at giving me right answers. Almost two years ago I don't believe they had every python package indexed and now they appear to have at least the documentation or source code of it.

XorNot• 1 year ago

The trouble is the only reliable use-case LLMs actually seem good at is "augmented search engine". Any attempts at coding with them just end up feeling like trying to code via a worse interface.

So it's handy to get a quick list of "all packages which do X", but it's worse then useless to have it speculate as to which one to use or why, because of the hallucination problem.

zitterbewegung2• 1 year ago

Yes it does work as an augmented search engine but, it does output working code you just have to prompt it better if the code is not that complex like a simple endpoint you just have to understand exactly what you want.

anigbrowl• 1 year ago

I've found them tobe quite a time saver, within limits. The blog post seemed scattered and disorganized to me, and the author admits having no experience with using LLMs to this end, so perhaps the problem lies behind their eyes.

addaon• 1 year ago

There's not much actual LLM-generated text in this post to go by, but it seems like each of the tokens generated by the LLM would be reasonable to have high probability. It sounds like the developer here thought that the sequence of tokens then carried meaning, where instead any possible meaning came from the reading. I wonder if this developer would be as irritated by the inaccuracies if they had cast sticks onto the ground to manage their stock portfolio and found the prophecy's "meaning" to be plausible but inaccurate.

pieix2• 1 year ago

AI isn’t a co-pilot; it’s a junior dev faking competence. Trust it at your own risk.

This is a good take that tracks with my (heavy) usage of LLMs for coding. Leveraging productive-but-often-misguided junior devs is a skill every dev should actively cultivate!

codr72• 1 year ago

What you're doing is sacrificing learning for speed.

Which is fine, if it's a conscious choice for yourself.

piva002• 1 year ago

I don't think GP was talking about themselves being a junior using LLMs, at least my interpretation was that devs should learn how to leverage misguided junior, and LLMs are more-or-less on the level of a misguided junior.

Which I completely agree, I use LLMs for the cases where I do know what I'm trying to do, I just can't remember some exact detail that would require reading documentation. It's much quicker to leverage a LLM rather than going on a wild goose chase of the piece of information I know exists.

Also it's a pretty good tool to scaffold the boring stuff, asking a LLM "generate test code for X asserting A, B, and C" and editing it to be a proper test frees up mental space for more important stuff.

I wouldn't trust a LLM to generate any kind of business logic-heavy code, instead I use it as a quite smart template/scaffold generator.

nrb• 1 year ago

Leveraging productive-but-often-misguided junior devs is a skill every dev should actively cultivate!

Feels like this is only worthwhile because the junior dev learns from the experience; an investment that yields benefits all around, in the broad sense. Nobody wants a junior around that refuses to learn in perpetuity, serving only as a drag on productivity and eventually your sanity.

simonw4• 1 year ago

That's somewhere that the AI-as-junior-dev analogy breaks down a little.

There's still incredible accumulated value here, but it's at the other end. The more times you successfully use an LLM to produce working code, the more you learn about how to use them - what they're good at, what they're bad at, how to effectively prompt them.

krageon• 1 year ago

It's worthwhile because you cannot do everything and often it is better to have someone far worse than you work on a problem than to just ignore it.

jdietrich2• 1 year ago

It is quite remarkable that we are already at the stage where saying "this AI is about as competent as an inexperienced college graduate" constitutes criticism. It is entirely proper for people to be engaging sceptically with LLMs at their current level of capability, but I think we should also keep in mind the astonishingly rapid growth rate in their performance. LLMs were a toy two years ago, they're now a useful if flawed colleague, but what can we expect two years from now?

acedTrex2• 1 year ago

I mean 2 years ago they were at about the same place, theres been very little practical gain from gpt4 in my opinion. No matter the model the fundamental failure cases have remained the same.

stuaxo• 1 year ago

The junior dev faking competence is useful but needs a lot of supervision (unlike a real junior dev we don't know if this one will get better).

tacoooooooo• 1 year ago

the "AI lies" takeaway is way off for those actually using these tools. Calling it a "junior dev faking competence" is catchy, but misses the point. We're not expecting a co-pilot, it's a tool, a super-powered intern that needs direction. The spaghetti code mess wasn't AI "lying", it was a lack of control and proper prompting.

Experienced folks aren't surprised by this. LLMs are fast for boilerplate, research, and exploring ideas, but they're not autonomous coders. The key is you staying in charge: detailed prompts, critical code review, iterative refinement. Going back to web interfaces and manual pasting because editor integration felt "too easy" is a massive overcorrection. It's like ditching cars for walking after one fender bender.

Ultimately, this wasn't an AI failure, it was an inexperienced user expecting too much, too fast. The "lessons learned" are valid, but not AI-specific. For those who use LLMs effectively, they're force multipliers, not replacements. Don't blame the tool for user error. Learn to drive it properly.

cruffle_duffle2• 1 year ago

“Experienced folks” in this case means folks who’ve used LLM’s enough to somewhat understand how to “feed them” in ways that make the tools generate productive output.

Learning to properly prompt an LLM to get a net gain in value is a skill in it of itself.

latexr2• 1 year ago

We're not expecting a co-pilot

Microsoft’s offering is literally called “copilot”. That is exactly what they’re marketing it as.

potsandpans• 1 year ago

Counterexample: Ive been able to complete more side projects in the last month leveraging llms than i have ever in my life. One of which I believe to have potential as a viable product, and another which involved complicated rust `no_std` and linker setup for compiling rust code onto bare metal RISCV from scratch.

I think the key to being successful here is to realize that you're still at the wheel as an engineer. The llm is there to rapidly synthesize the universe of information.

You still need to 1) have solid fundamentals in order to have an intuition against that synthesis, and 2) be experienced enough to translate that synthesis into actionable outcomes.

If youre lacking in either, youre at the same whims of copypasta that have always existed.

mythrwy2• 1 year ago

I've had both experiences strangely enough.

talldayo3• 1 year ago

which involved complicated rust `no_std` and linker setup for compiling rust code onto bare metal RISCV from scratch.

That's complicated, but I wouldn't say the resulting software is complex. You gave an LLM a repetitive, translation-based job, and you got good results back. I can also believe that an LLM could write up a dopey SAAS in half the time it would take a human to do the same.

But having the right parameters only takes you so far. Once you click generate, you are trusting that the model has some familiarity with your problem and can guide you without needing assistance. Most people I've seen rely entirely on linting and runtime errors to debug AI code, not "solid fundamentals" that can fact-check a problem they needed ChatGPT to solve first place. And the "experience" required to iterate and deploy AI-generated code basically boils down to your copy-and-paste skills. I like my UNIX knowledge, but it's not a big enough gate to keep out ChatGPT Andy and his cohort of enthusiastic morons.

We're going to see thousands of AI-assisted success stories come out of this. But we already had those "pennies on the dollar" success stories from hiring underpaid workers out of India and Pakistan. AI will not solve the unsolved problems of our industry and in many ways it will exacerbate the preexisting issues.

HPsquared• 1 year ago

What it will do is to free up a lot of brainpower to think about those hard problems and empower people to try our their ideas.

simonw4• 1 year ago

A tool that can "write up a dopey SAAS in half the time it would take a human to do" is a pretty incredible thing to add to your toolbox!

baxtr• 1 year ago

Is it reasonable to assume that more senior devs benefit more from LLMs?

dogma11382• 1 year ago

It depends it think it’s less about how senior they are and how good they are at writing requirements, and knowing what directives should be explicitly stated and what can be safely inferred.

Basically if they are good at utilizing junior developers and interns or apprentices they probably will do well with an LLM assistant.

adamtaylor_13• 1 year ago

I believe so. In my experience, you need to have that gut intuition (or experience) to say, “No way. That’s totally wrong.”

Since AI will capitulate and give you whatever you want.

You also have to learn how to ask without suggesting because it will take whatever you give it and agree.

jumpman500• 1 year ago

Ya. I think people that have better technical vocabulary and an understanding of what should be possible with code do better. That’s usually a senior engineer, but not always

RickS• 1 year ago

IMO experience provides better immunity for common hangups. Generated code tends to be directionally pretty good, but with lots of minor/esoteric failures. The experience to spot those fast and tidy them makes all the difference. Copilot helps me move 10x faster with tedious arduino stuff, but I can easily see where if I didn't have decent intuition around debugging and troubleshooting, there'd be almost zero traction since it'd be hard to clear that last 10% hurdle needed to even run the thing.

Spivak• 1 year ago

Yes, you essentially have an impossibly well read junior engineer you can task with quick research questions like, "I'm trying to do x using lib y, can you figure that out for me." This is incredibly productive because in the answer is typically all the pieces you need but not always assembled right.

Getting the LLM to pull out well-known names of concepts is for me the skill you can't get anywhere else. You can describe a way to complete a task and ask for what it's called and you'll be heading down arxiv links right away. Like yes the algorithm to find the closest in edit distance and length needle string in a haystack is called Needleman–Wunsch, of course Claude, everyone knows that.

simonw4• 1 year ago

I think so.

Junior devs can get plenty of value out of them too, if they have discipline in how they use them - as a learning tool, not as a replacement for thinking about projects.

Senior devs can get SO much more power from these things, because they can lean on many years of experience to help them evaluate if the tool is producing useful results - and to help them prompt it in the most effective way possible.

A junior engineer might not have the conceptual knowledge or vocabulary to say things like "write tests for this using pytest, include a fixture that starts the development server once before running all of the tests against it".

ern• 1 year ago

I had a couple of the most capable senior developers reach out to me to tell me how Github Copilot accelerated their productivity, which surprised me initially. So I think there's something to it.

jaredcwhite• 1 year ago

I wouldn't assume that at all. Most of the senior devs I talk to on a regular basis think commercial* LLMs are ridiculous and the AI hype is nonsensical.

* I put commercial there as a qualifier because there's some thought that in the future, very specifically-trained smaller models (open source) on particular technologies and corpuses (opt-in) might yield useful results without many of the ethical minefields we are currently dealing with.

satvikpendem4• 1 year ago

It's the LLM paradox, seniors get more senior with them while juniors get more junior, creating a bimodal distribution in the future simply because juniors will start depending on them too much to learn how to code properly while seniors (who some may also exhibit the previous trait) will by and large be able to rapidly synthesize information from LLMs with their own understanding.

dogma11382• 1 year ago

Indeed LLMs are useful as an intern, they are at the “cocky grad” stage of their careers. If you don’t understand the problem and can’t steer the solution and worse has only limited understanding of the code they produce you are unlikely to be productive.

On the other hand if you understand what needs to be done, and how to direct the work the productivity boost can be massive.

Claude 3.5 sonnet and O1 are awesome at code generation even with relatively complex tasks and they have a long enough context and attention windows that the code they produce even on relatively large projects can be consistent.

I also found a useful method of using LLMs to “summarize” code in an instructive manner which can be used for future prompts. For example summarizing a large base class that may be reused in multiple other classes can be more effective than having to overload a large part of your context window with a bunch do code.

pfannkuchen2• 1 year ago

I’ve found LLMs to basically be a more fluent but also more lossy way of interfacing with stack overflow and tutorials.

If a topic is well represented in those places, then you will get your answer quicker and it can be to some extent shaped to your use case.

If the topic is not well represented there, then you will get circular nonsense.

You can say “obviously, that’s the training data”, and that’s true, and I do find it obvious personally, but the reaction to LLMs as some kind of second coming does not align with this reality.

biohcacker84• 1 year ago

That matches my experience too. I wonder how fast they'll improve and if LLMs will hit a wall, as some AI experts think.

maybesomaybenot7• 1 year ago

Claude is like having my own college professor. I've learned more in the past month with Claude then I learned in the past year. I can ask questions repeatedly and get clarification as fine as a need it. Granted, Claude has limits, but its a game-changer.

I think the key to being successful here is to realize that you're still at the wheel as an engineer. The llm is there to rapidly synthesize the universe of information.

Bingo. OP is like someone who is complaining about the tools, when they should be working on their talent. I have a LOT of hobbies (circuits, woodworking, surfing, playing live music, cycling, photography) and there will always be people who buy the best gear and complain that the gear sucks. (NOTE: I"m not implying claude is "the best gear", but it's a big big help.)

I think the only problem with LLMs is synthesis of new knowledge is severely limited. They are great at explaining things others have explained, but suck hard at inventing new things. At least that's my experience with Claude: it's terrible as a "greenfield" dev.

XenophileJKO• 1 year ago

I would add though, that they can be very good at combining known concepts. Which can create a non-trivial set of "new knowledge".

KerrAvon• 1 year ago

How do you know it's accurate?

BalinKing3• 1 year ago

Claude is like having my own college professor.

I don't use Claude, so maybe there's a huge gap in reliability between it and ChatGPT 4o. But with that disclaimer out of the way, I'm always fairly confused when people report experiences like these—IME, LLMs fall over miserably at even very simple pure math questions. Grammatical breakdowns of sentences (for a major language like Japanese) are also very hit-or-miss. I could see an LLM taking the place of, like, an undergrad TA, but even then only for very well-trod material in its training data.

(Or maybe I've just had better experiences with professors, making my standard for this comparison abnormally high :-P )

EDIT: Also, I figure this sort of thing must be highly dependent on which field you're trying to learn. But that decreases the utility of LLMs a lot for me, because it means I have to have enough existing experience in whatever I'm trying to learn about so that I can first probe whether I'm in safe territory or not.

EarthMephit• 1 year ago

I find that LLMs are almost comically bad at projects that have a hardware component like RaspberryPi or Pico, or Ardunio.

I think that its because often the libraries you use are niche or have a a few similar versions, the LLM really commonly hallucinated solutions and would continually suggest that library X did have that capability. I think because often in hardware projects you often hit a point where you can't do something or you need to modify a library, but the LLM tries to be "helpful" and it makes up a solution.

chamomeal2• 1 year ago

They’re also pretty bad at typescript generics. They’re quite good at explaining concepts (like mapped types), but when push comes to shove they generate all sorts of stuff that looks convincing, but doesn’t pass the type checker.

And then you’ll paste in the error, and they’ll just say “ok I see the problem” and output the exact same broken code lol.

I’m guessing the problem is lack of training data. Most TS codebases are mostly just JS with a few types and zod schemas. All of the neat generic stuff happens in libraries or a few utilities

jerf• 1 year ago

Based on my own modestly successful forays into that world, I have to imagine one problem the LLMs have in that space is terrible training data. A good three quarters of any search result you search for in that space will be straight-up out-of-date and not work on your system. Then you've got all the tiny variations between dozens of chipsets, and all the confidently wrong people on the internet telling you to do nonsensical things, entire ecosystems that basically poofed out of existence three years ago but are still full of all kinds of juicy search terms and Google juice... if I can hardly paw through this stuff with years of experience and the physical hardware right in front of me to verify claims with, I don't know how an LLM is supposed to paw through all that and produce a valid answer to any question in that space, without its own hardware to fiddle with directly.

swatcoder• 1 year ago

Actually, it's because many of the people writing tutorials and sharing answers about that stuff don't know what the hell they're doing or grasp the fundamentals of how those systems work and so most of the source material the LLM's are trained on is absolute garbage.

Public Arduino, RPi, Pico communities are basically peak cargo cult, with the blind leading the blind through things they don't understand. The noise is vastly louder than the signal.

There's a basically giant chasm between expereinced or professional embedded developers that mostly have no need to ever touch those things or visit their forums, and the confused hobbyists on those forums randomly slapping together code until something sorta works while trying to share their discoveries.

Presumably, those communities and their internal knowledge will mature eventually, but it's taking a long long time and it's still an absolute mess.

If you're genuinely interested in embedded development and IoT stuff, and are willing to put in the time to learn, put those platforms away and challenge yourself to at least learn how to directly work with production-track SoC'a from Nordic or ESP or whatever. And buy some books or take some courses instead of relying on forums or LLM's. You'll find yourself rewarded for the effort.

jrmg• 1 year ago

I was just today trying to fix some errors in an old Linux kernel version 3.x.x .dts file for some old hardware, so that I could get a modern kernel to use it. ChatGPT seemed very helpful at first - and I was super impressed. I thought it was giving me great insight into why the old files were now producing errors … except the changes it proposed never actually fixed anything.

Eventually I read some actual documentation and realised it was just spouting very plausible sounding nonsense - and confident at it!

The same thing happened a year or so ago when I tried to get a much older ChatGPT to help me with with USB protocol problems in some microcontroller code. It just hallucinated APIs and protocol features that didn’t actually exist. I really expected more by now - but I now suspect it’ll just never be good at niche tasks (and these two things are not particularly niche compared to some).

onemoresoop• 1 year ago

I find LLMs to be decent unblockers. I only turn to them form time to time though, unless Im in a playful mode and try poking out various ideas. As a coder I also ask for snippets when Im lazy. I tried growing a slightly larger solution a few times and it failed in dumb ways. It was clear it doesn’t really comprehend we do, it’s not aware it’s moving in circles and so on. All these things will probably see a lot incremental of improvents and as a tool will definitely stay but fundamentally LLMs can’t really think, at least the way we do and expecting that is also foolish.

chamomeal2• 1 year ago

I agree with his point about asking AI to “fix” problems though. It’s really nice that you don’t have to fully understand something to use it, but that becomes a problem if you lean on it too much

gsf_emergency_2• 1 year ago

Ime engineers who find LLM useful have misunderstood their reasons for existence outside of being salaried..

What is your main project ? Do you LLM that? (

I wager you're not a rust expert and should maybe reconsider using rust in your main project.

FWIW asking LLM whether you should use rust ~ asking it about the meaning of life. Important questions that need answers, but not right away! (A week or 2 tops)

If you need to synthesize the universe of information with LLM.. that is not the universe you want to live or play in

blast• 1 year ago

The llm is there to rapidly synthesize the universe of information

That's a nice way of putting it.

pinoy420• 1 year ago

Exactly this. OP, credit where credit is due, appears to be someone who “hacks things together” copy pasting solutions blindly from the internet - with little intuition gained along the way.

irthomasthomas• 1 year ago

Another data point. Plugin ideation to publication in 2 minutes, from a single prompt, albeit a multi-step shell prompt. 3 packages published in 24 hrs https://x.com/xundecidability/status/1884077427871342955

zackproser• 1 year ago

This

nsavage• 1 year ago

Funny enough, I posted an article I wrote here yesterday with the same sort of thesis. Different technologies (mine was Docker) but same idea of LLM leading me astray and causing a lot of frustration

powerset• 1 year ago

I've had a similar experience, shipping new features at incredible speed, then waste a ton of time going down the wrong track trying to debug something because the LLM gave me a confidently wrong solution.

williamcotton• 1 year ago

Well that's kind of on you for not noticing that it was the wrong solution, isn't it?

cruffle_duffle2• 1 year ago

I think the parents post happened to everybody, and if it hasn’t it will.

The edge between being actually more productive or just “pretend productive” using large language models is something that we all haven’t completely figured out yet.

trinix912• 1 year ago

Sometimes the solution is 99% correct but the other 1% is so subtly wrong that it both doesn't work and is a debugging hell.

mythrwy2• 1 year ago

Ya but you kind of get painted in corner sometimes. And sunken cost fallacy kicks in.

nyarlathotep_• 1 year ago

often it's something you casually overlook, some minor implementation detail that you didn't give much thought to that ends up being a huge mess later on, IME

th0ma5• 1 year ago

It's on you however for not understanding the greater point?

cherry_tree• 1 year ago

Seems like LLMs would be well suited for test driven development. A human writes tests and the LLM can generate code passing all tests; ending with a solution that meets the humans expectations.

drpossum• 1 year ago

I disagree because you're only considering the "get code to make the test pass". Refactoring, refining, and simplifying is critical and I've yet to see this applied well. (I've also yet to see the former applied usably well either despite "write tests generate code" being an early direction.)

distortionfield• 1 year ago

This is more or less how I use LLMs right now. They’re fantastic at the plumbing, so that I can focus on the important part - the business and domain logic.

KronisLV2• 1 year ago

In my experience LLMs will help you with things that have been solved thousands of times before and are just a matter of finding some easily researched solution.

The very moment when you try to go off the beaten path and do something unconventional or stuff that most people won't have written a lot about, it gets more tricky. Just consider how many people will know how to configure some middleware in a Node.js project... vs most things related to hardware or low level work. Or even working with complex legacy codebases that have bits of code with obscure ways of interacting and more levels of abstraction that can be reasonably put in context.

Then again, if an LLM gets confused, then a person might as well. So, personally I try to write code that'd be understandable by juniors and LLMs alike.

winocm• 1 year ago

In my experience, a LLM decided to not know type alignment rules in C and confidently trotted out the wrong answer. It left a horrible taste in my mouth for the one time I decided to look at using a LLM for anything and it keeps leaving me wondering if I'd end up more time bashing the LLM into working than just working out the answer myself and learning the underlying reasoning why.

It was so wrong that I wonder what version of the C standard it was even hallucinating.

NitpickLawyer2• 1 year ago

vs most things related to hardware or low level work.

counter point:

https://github.com/ggerganov/llama.cpp/pull/11453

This PR provides a big jump in speed for WASM by leveraging SIMD instructions for qX_K_q8_K and qX_0_q8_0 dot product functions.
Surprisingly, 99% of the code in this PR is written by DeekSeek-R1. The only thing I do is to develop tests and write prompts (with some trials and errors)

alfalfasprout2• 1 year ago

A single PR doesn't really "prove" anything. Optimization passes on well-tested narrowly scoped code are something that LLMs are already pretty good at.

suddenlybananas3• 1 year ago

at a certain point though, one wonders if you can trust people to accurately report how much is written by an LLM. (not even implying bad faith, but if you're constantly re-reading, selecting and re-combining snippets written by LLMs, it's not really "written" by LLMs in the same way that's implied).

ryandrake• 1 year ago

I use CoPilot pretty much as a smarter autocomplete that can sometimes guess what I'm planning to type next. I find it's not so good at answering prompts, but if I type:

    r = (rgba >> 24) & 0xff;

...and then pause, it's pretty good at guessing:

    g = (rgba >> 16) & 0xff;
    b = (rgba >> 8) & 0xff;
    a = rgba & 0xff;

... for the next few lines. I don't really ask it to do more heavy lifting than that sort of thing. Certainly nothing like "Write this full app for me with these requirements [...]"

schmookeeg• 1 year ago

Brilliant.

I have fought the "lowest cognitive load" code-style fight forever at my current gig, and I just keep losing to the "watch this!" fancytowne code that mids love to merge in. They DO outnumber me, so... fair deuce I suppose.

There is value in code being readable by Juniors and LLMs -- hell, this Senior doesn't want to spend time figuring out your decorators, needless abstractions and syntax masturbation. I just want to fix a bug and get on with my day.

KronisLV2• 1 year ago

While I think this comment got flagged (probably for the way it was worded), you aren't wrong! A good way I've heard a similar thought expressed is that code should be not only easy to maintain, but also throw away and replace, which more or less urges you towards writing the easiest things you can get away with (given your particular performance requirements and other constraints).

rooroobooragool• 1 year ago

That rhymes with my experience of trying to generate placeholder art using AI.

Since it's just a placeholder I often ask for a funny twist but it's rarely ever anything like it.

mercer• 1 year ago

I've only started using LLMs for code recently, and I already tend to mentally translate what I want to something that I imagine is 'more commonly done and well represented in the training data'.

But especially the ability to just see some of the stuff it produces, and now to see its thought process, is incredibly useful to me already. I do have autism and possibly ADD though.

tasuki• 1 year ago

LLMs are surprisingly good at Haskell (and I'm not).

I hope for a rennaisance of somewhat more rigorous programming languages: you can typecheck the LLM suggestions to see if they're any good. Also you can feed the type errors back to the LLM.

anaisbetts• 1 year ago

Context matters a lot, copy-pasting snippets to a webpage is _way_ less effective than Cursor/Windsurf.

cmdtab2• 1 year ago

Today, I needed to write a proxy[0] that wraps an object and log all method calls recursively.

I asked claude to write the initial version. It came up with a complicated class based solution. I spent more than 30 minutes getting a good abstract to come out. I was copy pasting typescript errors and applying fixes it suggested without thinking much.

In the end, I gave up and wrote what I wanted myself in 5 minutes.

0] https://github.com/cloudycotton/browser-operator/blob/main/s...

dambi03• 1 year ago

Would you have written it in five minutes has you not just spent 30 minutes ruling out wrong solutions?

cmdtab2• 1 year ago

Yes. I wrote how to do it technically. Claude was able to come up with a solution that worked on second attempt. The problem was it didn’t work with typescript nicely. The approach overcomplicated anything that depended on this class.

transcriptase• 1 year ago

I’ve been able to do more far complex things with ESP32s and RPis in an evening without knowing the first thing about python or c++.

I can also tell when it’s stuck in some kind of context swamp and won’t be any more help, because it will just keep making the same stupid mistakes over and over and generally forgetting past instructions.

At that point I take the last working code and paste it into a new chat.

thot_experiment• 1 year ago

As opposed to not trusting an LLM, and ending up on day 4 of an afternoon project? :P

I've been doing that since way before LLMs were a thing.

ravroid2• 1 year ago

One strategy I've been experimenting with is maintaining a 'spec' document, outlining all features and relevant technical notes about a project. I include the spec with all relevant source files in my prompt before asking the LLM to implement a new change or feature. This way it doesn't have to do as much guessing as to what my code is doing, and I can avoid relying on long-running conversations to maintain context. Instead, for each big change I include an up-to-date spec and all of the relevant source files. I update the spec to reflect the current state of the project as changes are made to give the LLM context about my project (this also doubles as documentation).

I use an NPM script to automate concatenating the spec + source files + prompt, which I then copy/paste to o1. So far this has been working somewhat reliably for the early stages of a project but has diminishing returns.

SparkyMcUnicorn• 1 year ago

You're describing functionality that's built into Aider. You might want to try it out.

Aider also has a copy/paste mode to use web ui interfaces/subscriptions instead apis.

I definitely use and update my CONVENTIONS.md files and started adding a second specification file for new projects. This + architect + "can your suggestion be improved, or is there a better way?" has gotten me pretty far.

ravroid2• 1 year ago

Didn't know about Aider - going to give that a try, thanks!

mordymoop4• 1 year ago

I am frankly tired of seeing this kind of post on HN. I feel like the population of programmers is bifurcating into those who are committed to mastering these tools, learning to work around their limitations and working to leverage their strengths… and those who are committed to complaining about how they aren’t already perfect Culture Ship Minds.

We get it. They’re not superintelligent at everything yet. They couldn’t infer what you must’ve really meant in your heart from your initial unskillful prompt. They couldn’t foresee every possible bug and edge case from the first moment of conceptualizing the design, a flaw which I’m sure you don’t have.

The thing that pushes me over the line into ranting territory is that computer programmers, of all people, should know that computers do what you tell them to.

antigeox• 1 year ago

I am frankly tired of seeing this kind of post on HN.

You've been here since 2016 and this is the kind of posting that finally gets to you? How in the world have you avoided all the shitposts in the last decade? What is your secret?

mordymoop4• 1 year ago

It’s the fact that it’s the same thing over and over. The 100th case of “Well I had a bad experience using LLM for coding!” is not interesting.

bigstrat2003• 1 year ago

Nobody is complaining that LLMs aren't perfect Culture minds. People disagree with the premise that they are useful tools given their current capabilities. Your portrayal of those with whom you disagree is such a strawman that it might as well be set to a soy-vs-wojak meme.

mordymoop4• 1 year ago

They clearly are useful tools given their current capabilities. It just depends on what you’re using them for. You don’t use a screwdriver to drive nails, and you don’t go to HardwareNews to complain when your screwdriver isn’t working as a hammer.

I’m currently using them to port a client-side API SDK into multiple languages. This would be a pain in the ass time consuming task but is a breeze with LLMs because the exact behavior I want is clearly defined and relatively deterministic, and it’s also straightforward to test that I’m getting what I intend. The LLM thus gets done in 3 days what would take me 3 weeks (or more) to do by hand.

If the complaint is that it can’t do X, where X is something that would clearly require full AGI and likely true superintelligence — in this case expecting instantaneous, correct code that solves novel problems on the first try - then I have to insist that people are actually expecting Claude to be a Culture Ship Mind, implicitly. They just don’t realize that what they’re asking for his hard, which is itself a psychologically interesting fact, I suppose.

simonw4• 1 year ago

"People disagree with the premise that they are useful tools given their current capabilities."

I will argue the opposite of that forever. They're very evidently useful, if you take the time to learn how to apply them.

suddenlybananas3• 1 year ago

The thing that pushes me over the line into ranting territory is that computer programmers, of all people, should know that computers do what you tell them to.

are you claiming LLMs function like computer program instructions? like they clearly don't operate like that at all.

dambi03• 1 year ago

What part of the comment makes you think they are claiming that?

mordymoop4• 1 year ago

They are closer to being deterministic machines that comply exactly with your instructions, for better or worse, than they are to magical pixies that guess what you must’ve actually meant. The implicit expectation demonstrated by many in the “loudly disappointed in LLMs” contingent seems to be that LLMs should just know what you meant, and then blame them for not correctly guessing it and delivering it.

I think LLMs have uncovered what we have always known in this industry: that people are, by default, bad at communicating their intent clearly and unambiguously.

If you express your intent to an LLM with sufficient clarity and disambiguation, it will rarely screw up. Often, we don’t have time to do this, and instead we aim for the sweet spot of sufficient but not exhaustive clarity. This can be fine if you are experienced with that particular LLM and you have a good feel for where its sweet spot actually is. If you miss that target, though, the LLM will not correctly infer your intended subtext. This is one of the things that requires experience. In fact, even the “same” LLM will change in its behavior and capabilities as it undergoes fine tuning. Sometimes it will even get worse at certain things.

All of this is to say, of course, you’re right that it’s not a compiler. But I think people fail in their application of LLMs for much the same reason that novice coders fail to get compilers to guess what they intended.

rglover2• 1 year ago

computer programmers, of all people, should know that computers do what you tell them to.

Right. The problem isn't that the tool isn't perfect, it's that you get a lot of excitable people with incentives pretending that it is or will soon be perfect (while simultaneously scaring non-technical people into thinking they'll be replaced with a chat bot soon).

There are certainly luddite types who are outright rejecting these tools, but if you have hands-on, daily experience, you can see the forest for the trees. You quickly realize that all of the "omg this thing is sentient" or "we can't let what we've got into the world, it's too dangerous" fodder like the Google panic memo are just covert marketing.

satvikpendem4• 1 year ago

Use uBlock Origin to block posts with the keywords you don't want in the title, like AI or LLM.

rglover2• 1 year ago

AI is great for generating ideas or drafting code, but it doesn’t understand. It’s like giving a junior developer a chainsaw instead of a scalpel—it might finish the job, but you’ll spend twice as long cleaning up the mess.

For anything remotely complex, this is dead on. I use various models daily to help with coding, and more often than not, I have to just DIY it or start brand new chats (because the original context got overwhelmed and started hallucinating).

This is why it's incredibly frustrating to see VCs and AI founders straight-up gaslighting people about what this stuff can (or will) do. They're trying to push this as a "work killer," but really, it's going to be some version of the opposite: a mess creator that necessitates human intervention.

Where we're at is amazing, but we've got a loooong way to go before we can be on hover crafts sipping sodas Wall-E style.

lxe• 1 year ago

I'm developing an intuition to how and what to ask in order for the LLM's answer to be helpful. Once you start spinning your wheels, clear context, copy what you need, and start over.

insane_dreamer2• 1 year ago

junior dev faking competence

A bit on a tangent, but has there been any discussion of how junior devs in the future are ever going to get past that stage and become senior dev calibre if companies can replace the junior devs with AIs? Or is the thinking we'll be fine until all the current senior devs die off and by then AI will be able to replace them too so we won't need anyone?

1. CS/Eng degree 2. ??? 3. Senior dev!

dragonwriter2• 1 year ago

Training will adapt to the widespread use of AI coding assistance if they are that universally useful, and people will come into the market as junior AI wranglers, with skillsets stronger than curent junior devs is some areas but weaker in others; current seniors will grumble about holes in their knowledge, but that's been the case with the generational changes in software development as the common problems people face at different levels have shifted over time. The details are new, but the process isn't.

insane_dreamer2• 1 year ago

Not if the goal is to replace the junior devs with AIs -- people won't be "coming into the market" because they won't be needed.

Companies are not saving money by paying for AI tools if they continue to hire the same number of people. The only way it makes financial sense, and for the enormous amounts of money being invested into AI to reap profits, is if companies are able to reduce the cost of labor. First, they only need 75% of the junior devs they have now, then 50%, then 25%.

wvenable• 1 year ago

It's definitely as good as junior dev at a lot of tasks but you always have to be in the driver seat. I don't ask junior devs to write functions one at a time. I give them a task, they ping me if they need something, but otherwise I hope I don't hear from them again for a while.

I don't see AI replacing that. AI is a tool with the instant Q&A intelligence of a junior dev but it's not actually doing the job of a junior dev. That's a subtle distinction.

dstainer• 1 year ago

One of the areas where I've struggled to get effective use out of the LLM's is with UI/UX. That isn't my primary area of expertise (backend) so it definitively could be operator error here, but I use tools like v0.dev and just can't quite get it to do what I need it to do. Anybody have any tools, workflows, suggestions for this?

DigitalSea• 1 year ago

This is like watching a carpenter blame their hammer because they didn’t measure twice. AI is a tool, it's like a power tool for a tradesperson: it'll amplify your skills, but if you let it steer the whole project? You’ll end up with a pile of bent nails.

LLMs are jittery apprentices. They'll hallucinate measurements, over-sand perfectly good code, or spin you in circles for hours. I’ve been there back in the GPT-4 days especially, nothing stings like realising you wasted a day debugging AI’s creative solution to a problem you could've solved in 20 minutes.

When you treat AI like a toolbelt, not a replacement for your own brain? Magic. It’s killer at grunt work like; explaining regex, scaffolding boilerplate, or untangling JWT auth spaghetti. You still gotta hold the blueprint. AI ain't some magic wand: it’s a nail gun. Point it wrong, and you’ll spend four days prying out mistakes.

Sucks it cost you time, but hey, now you know to never let the tool work you. It's hopefully a lesson OP learns once and doesn't let it sour their experience with AI, because when utilised properly, you can really get things done, even if it's just the tedious/boring stuff or things you'd spend time Google bashing, reading docs or finding on StackOverflow.

groby_b• 1 year ago

I mean....

No design. Hardware & software. 2 different platforms. A new language. Zig. Unrealistic time expectations.

A senior SWE would've still tanked this, just in different ways.

Personally, I'd still consider it a valuable experiment, because the lessons learned are really valuable ones. Enjoy round 2 :)

protocolture• 1 year ago

I used very similar hardware to accomplish a very similar project (Notifications on a round screen) and the LLM was great for everything except UX.

tippytippytango• 1 year ago

What I learned is you can’t outsource expertise to an LLM, after many similar experiences to OP. Don’t ask it for advice, ask it to teach you so you can make decisions like this on your own. Preferably ground questions with excerpts from human made documents. It seems to make less mistakes when explaining things, and those mistakes are more noticeable when they do happen.

Fourier864• 1 year ago

FWIW, I fed in the same problematic prompt to all the current ChatGPT models and even the legacy/mini models enumerated a bunch of pitfalls and considerations. I wonder why/how it managed to tell the author everything was perfect? A weird one-off occurrence?

theodric• 1 year ago

"From her perspective, I order doordash and turn into a degen who is unfit to father. From my perspective, I get to enjoy my favorite place and just tinker or play games or do whatever. These are the days I get to play mad scientist and feel most like myself."

Most demeaning and depressingly toxic thing I've read today...

neilv• 1 year ago

TLDR - AI isn’t a co-pilot; it’s a junior dev faking competence. Trust it at your own risk.

A junior dev faking competence while plagiarizing like crazy.

The plagiarizing part is why the junior dev from hell might not get fired: laundering open source copyrights can have beancounter alignment.

65• 1 year ago

I have never found a use for LLMs for programming because I can find the (correct) answer much easier with a search. Perhaps the search engines just suck so hard these days people resort to LLMs. I use Kagi and GitHub to search and the results are much better.

turnsout• 1 year ago

Counterpoint: I'm on day 26 of an afternoon project I never would have attempted on my own, and I'm going to release it as a major feature.

Cursor & Claude got the boilerplate set up, which was half the mental barrier. Then they acted as thought partners as I tried out various implementations. In the end, I came up with the algorithm to make the thing performant, and now I'm hand-coding all the shader code—but they helped me think through what needed to be done.

My take is: LLMs are best at helping you code at the edge of your capabilities, where you still have enough knowledge to know when they're going wrong. But they'll help you push that edge forward.

lbotos• 1 year ago

Nemo, are you using a self-hosted install or .com?

If .com, email me (it's in my profile) and I can see if there is a reason your account is getting so heavily captcha'd.

adsharma• 1 year ago

A few mitigation ideas

  * Get AI to write tests
  * Use copy/paste. No IDE
  * Use python (not because it's better than zig)

tashian2• 1 year ago

I used Claude to help me build a side project in 4 hours that I would never have built otherwise. Essentially, it's a morphing wavetable oscillator in React (https://waves.tashian.com).

Six months ago, I tried building this app with ChatGPT and got nowhere fast.

Building it with Claude required a gluing together a few things that I didn't know much about: JavaScript audio processing, drawing on a JavaScript canvas, an algorithm for bilinear interpolation.

I don't write JavaScript often. But I know how to program and I understand what I'm looking at. The project came together easily and the creative momentum of it felt great to me. The most amazing moment was when I reported a bug—I told Claude that the audio was stuttering whenever I moved the controls—and it figured out that we needed to use an AudioWorklet thread instead of trying to play the audio directly from the React component. I had never even heard of AudioWorklet. Claude refactored my code to use the AudioWorklet, and the stutter disappeared.

I wouldn't have built this without Claude, because I didn't need it to exist that badly. Claude reduced the creative inertia just enough for me to get it done.

masklinn• 1 year ago

What was your workflow for doing that? Just going back and forth in a chat, or a more integrated experience in a dedicated editor?

tashian2• 1 year ago

Just copy/paste from the chat window. I kept running into token limits. I came away from it wanting a much better workflow.

That's the next step for me in learning AI... playing with different integrated editor tools.

Havoc• 1 year ago

That’s a fun writeup.

I’ve come to a similar conclusion - for now at least it’s best applied at a fairly granular level. Make me a red brick wall there rather than „hey architect make me a house“.

I do think OP tried a bit too much new stuff in one go though. USB plus zig is quite a bit more ambitious than the traditional hello world in a new lang

rekabis• 1 year ago

As they say, I was trying to get a few birds stoned at once.

Imma gonna have to work this into a convo some day. Just to see the “wait, what??” expressions on people’s faces.

qiqitori• 1 year ago

Your plan was to use USB, but to me it looks like you're pretty much just using serial via USB. That's completely fine of course! One cheap way to tackle your problem is to use a version of printf with locking, which is likely available in many microcontroller SDKs (it's also slow). (Or you could add your own mutex.)

USB-CDC is cooler than that, you can make the Pico identify as more than just one device. E.g. https://github.com/Noltari/pico-uart-bridge identifies as two devices (so you get /dev/ttyACM0 and /dev/ttyACM1). So you could have logs on one and image transfers on another. I don't think you're limited to just two, but I haven't looked into it too far.

You can of course also use other USB protocols. For example you could have the Pico present itself as a mass-storage device or a USB camera, etc. You're just limited by the relatively slow speed of USB1.1. (Though the Pico doesn't exactly have a lot of memory so even USB1.1 will saturate all your RAM in less than 1 second)

medhir• 1 year ago

Perhaps being a PM for several years has helped, I’ve had great success speeding up my programming workflows by prompting Claude with very specific, well defined tasks.

Like many others are saying, you need to be in the drivers seat and in control. The LLM is not going to fully complete your objectives for you, but it will speed you up when provided with enough context, especially on mundane boilerplate tasks.

I think the key to LLMs being useful is knowing how to prompt with enough context to get a useful output, and knowing what context is not important so the output doesn’t lead you in the wrong direction.

Graziano_M• 1 year ago

OP could have written the firmware in zig, too!

https://github.com/ZigEmbeddedGroup/microzig

ffitch• 1 year ago

for what it’s worth, my afternoon projects tend to take over four days even if no llm is involved

raleighm• 1 year ago

Laughed aloud at this one. Yup, same here.

superq• 1 year ago

Most LLMs default to being sycophantic yes-men, but if you create a custom prompt, it can help mitigate any issues.

I have a custom prompt that instructs gpt4o to get aggressive about attacking anything I say (and, importantly, anything it says).

Here's my result for the same question:

https://chatgpt.com/share/67984aa9-1608-8012-be93-a77728ab8e...

cudgy• 1 year ago

Really enjoyed reading your article. Haven’t laughed as much reading a tech article in quite some time. You should consider doing some YouTube videos as your communications style is very humble and entertaining.

Made me wanna join in your garage and help out with the project :)

righthand• 1 year ago

I spend a good portion of my time asking people to fix their LLM code now at work. It has made code reviews tiring. And it has increased pairing time significantly, making it a less fun activity.

Joel_Mckay• 1 year ago

When workmanship doesn't matter, than the ship is already sinking.

It has been my experience 1 code clown can poison a project with dozens of reasonably talented engineers active. i.e. clowns often go through the project smearing bad kludges over acceptable standards to appear like their commit frequency means something.

This is why most developers secretly dream of being plumbers. Good luck, =3

j45• 1 year ago

I doubt one experience, positive, or negative counts for them all.

Learning how to build or create with a new kind of word processor is a skill unto itself.

petarb• 1 year ago

My wife and I have a deal where 1 day a month, She takes the kiddo and I am absolved of all responsibilities. I get a full day to lock in and build projects.

I love it!

djray• 1 year ago

The OP misunderstands (perhaps deliberately or for humorous effect) what a co-pilot is. This is telling:

"I learned that I need to stay firmly in the driver’s seat when tackling new tech."

Er, that's pretty much what a pilot is supposed to do! You can't (as yet) just give an AI free reign over your codebase and expect to come back later that day to discover a fully finished implementation. Maybe unless your prompt was "Make a snake game in Python". A pilot would be supervising their co-pilot at all times.

Comparing AIs to junior devs is getting tiresome. AIs like Claude and newer versions of ChatGPT have incredible knowledge bases. Yes, they do slip up, especially with esoteric matters where there are few authoritative (or several conflicting) sources, but the breadth of knowledge in and of itself is very valuable. As an anecdote, neither Claude nor ChatGPT were able to accurately answer a question I had about file operation flags yesterday, but when I said to ChatGPT that its answer wasn't correct, it apologised and said the Raymond Chen article it had sourced wasn't super clear about the particular combination I'd asked about. That's like having your own research assistant, not a headstrong overconfident junior dev. Yes, they make mistakes, but at least now they'll admit to them. This is a long way from a year or two ago.

In conclusion: don't use an AI as one of your primary sources of information for technology you're new to, especially if you're not double-checking its answers like a good pilot.

sarob• 1 year ago

You are running the show and the LLM can act like many other roles to help you. Obscure or confusing topics, likely the LLM will be as bad at solving as any employee. Give it a plan. Follow up and make sure it’s on track.

I trusted an LLM, now I'm on day 4 of an afternoon project

Related Stories