Why do LLMs still not run code before giving it to you?

https://news.ycombinator.com/item?id=44779238

by highfrequency • 12 months ago

1 3 12 months ago

news.ycombinator.com

The leading models all advertise tool use including code execution. So why is it still common to receive a short Python script containing a logical bug which would be immediately discoverable upon running a Python interpreter for 0.1 seconds? Is it a safety concern / difficulty sandboxing in a VM? Surely not a resource consumption issue given the price of a single CPU core vs. GPU.

chasing0entropy• 12 months ago

Sounds like an opportunity for you to make the world better by designing the process and implementing it.

tlb• 12 months ago

Is it a common use case to produce a standalone program that could be tested in isolation? Usually I'm asking for a function (or just a few lines of change) that depends on the rest of my code & environment, so it's not trivial to test.

serf• 12 months ago

depends on the methodology really.

if you're doing TDD style work but with an AI it's not uncommon to one-shot a function and then throw it against your battery of tests.

it's also pretty doable if you're writing smallish scripts or trying to follow functional coding paradigms; with functional stuff it's often easy to pick apart the specific modules for testing against criteria.

Why do LLMs still not run code before giving it to you?

Related Stories