An AI-first program synthesis framework built around a new programming language

https://queue.acm.org/detail.cfm?id=3746223

by tosh • 12 months ago

112 21 12 months ago

queue.acm.org

sys13• 12 months ago

Glad to see focus being put on keeping humans in the drivers seat, democratizing coding with the help of AI. The syntax is probably still too verbose to be easily accessible, but I like the overall approach.

AIPedant2• 12 months ago

I know ACM Queue is a non-peer-reviewed magazine for practitioners but this still feels like too much of an advertisement, without any attempt whatsoever to discuss downsides or limitations. This really doesn't inspire confidence:

  While this may seem like a whimsical example, it is not intrinsically easier or harder for an AI model compared to solving a real-world problem from a human perspective. The model processes both simple and complex problems using the same underlying mechanism. To lessen the cognitive load for the human reader, however, we will stick to simple targeted examples in this article.

For LLMs this is blatantly false - in fact asking about "used textbooks" instead of "apples" is measurably more likely to result in an error! Maybe the (deterministic, Prolog-style) Universalis language mitigates this. But since Automind (an LLM, I think) is responsible for pre/post validation, naively I would expect it to sometimes output incorrect Universalis code and incorrectly claim an assertion holds when it does not.

Maybe I am making a mountain out of a molehill but this bit about "lessen the cognitive load of the human reader" is kind of obnoxious. Show me how this handles a slightly nontrivial problem, don't assume I'm too stupid to understand it by trying to impress me with the happy path.

tannhaeuser6• 12 months ago

Prolog works indeed very well as target for generation by an LLM, for input problems limited and similar enough in nature to given classes of templated in-context examples, so well indeed that the lack of a succinct, exhaustive text description of your problem is becoming the issue. At which point you can specify your problem in Prolog directly considering Prolog was also invented to model natural language parsing and not just for solving constraint/logic problems, or could employ ILP techniques to learn or optimize Prolog solvers from existing problem solutions rather than text descriptions. See [1].

[1] https://quantumprolog.sgml.net

YeGoblynQueenne6• 12 months ago

> Since we're not doing original research, but rather intend to demonstrate a port of the Aleph ILP package to ISO Prolog running on Quantum Prolog, we cite the problem's definition in full from the original paper (ilp09):

Aleph? In 2025. That's just lazy, now. At the very least they should try Metagol or Popper, both with dozens of recent publications (and I'm not even promoting my own work).

AIPedant2• 12 months ago

Right but my point was that "given classes of templated in-context examples" is either

a) a game of roulette where you hope the LLM provider has RLHFed something very close to your use case, or

b) trying to few-shot it with in-context examples requires more engineering (and is still less reliable) than simply doing it yourself

In particular it's not just "the lack of a succinct, exhaustive text description," it also a lack of English->Prolog "translations."

It seems like the LLM-Prolog community is well aware of all this (https://swi-prolog.discourse.group/t/llm-and-prolog-a-marria...) but I don't see anything in Universalis that solves the problem. Instead it's just magically invoking the LLM.

duskwuff• 12 months ago

This article feels extremely imprecise. The syntax of the "language" changes from example to example, control structures like conditionals are expressed in English prose, some examples are solved by "do all the work for me" functions like the "toPdf()" example...

This whole thing feels like an elaborate LLM fantasy. Is there any real, usable language behind these examples, or is the author just role-playing with ChatGPT?

YeGoblynQueenne6• 12 months ago

More to the point, is ChatGPT role-playing the author?

mannycalavera42• 12 months ago

lol

ethanwillis• 12 months ago

Universalis ensures that even those with minimal experience in programming can perform advanced data manipulations.

Is it a good thing to make this easier? We're drowning in garbage already.

rvasa• 12 months ago

Great to start off ... then we will end-up reinventing/re-specifying functions for reusability, module/packages for higher-level grouping, types/classes, state machines, control-flows [with the nuances for edge cases and exit conditions], then we will need error control, exceptions; sooner or later concurrency, parallelism, data structures, recursion [lets throw in monads for the Haskellians amoung us]; who knows .. we may even end up with GOTOs peppered all over the English sentences [with global labels] & wake up to the scoping, parameter passing. We can have a whole lot of new fights if we need object-oriented programming; figure out new design patterns with special "Token Factory Factories".

We took a few decades to figure out how to specify & evolve current code to solve a certain class of problems [nothing is perfect .. but it seems to work at scale with trade-offs]. Shall watch this from a distance with pop-corn.

pjmlp• 12 months ago

So after Haskell, and helping with LINQ, this is what Erik Meijer has been focusing on.

gavinray• 12 months ago

  > "The actual implementation of Universalis uses Kotlin DataFrames"

Neat. My favorite talk at the 2025 Kotlin Conf was the demo of the new compiler plugin that allows ad-hoc type inference from DataFrame expressions.

Essentially, say that you have an input type like:

  type User { name: String, age: Int }

And you do something like ".map(it => { doubledAge: it.age * 2 })"

The inferred type of that intermediate operation is now:

  type User_synthetic { doubledAge: Int }

Which is wild, since you essentially have TypeScript-like inference in a JVM language.

Timestamp to talk:

https://www.youtube.com/watch?v=F5NaqGF9oT4&t=543s

An AI-first program synthesis framework built around a new programming language

Related Stories