Show HN: Program Synthesis for Ruby

by sankha93 • 4 years ago

82 19 4 years ago

eatonphil• 4 years ago

Given a method specification in the form of tests, type and effect annotations with RDL, this synthesizes a Ruby function that will pass the tests. This reduces programmer effort to just writing tests that specify the function behavior and the computer writing the function implementation for you.

That's really neat! But doesn't this just mean that instead of writing your code and logic in Ruby you're writing it in RDL?

Also, it would be great to see some examples in the README or in an examples folder. It's kind of hard for me to tell exactly what I need to write and what to call to generate the Ruby.

sankha938• 4 years ago

But doesn't this just mean that instead of writing your code and logic in Ruby you're writing it in RDL?

RDL allows the specification of types and effect labels only. You still write the tests in standard Ruby and synthesized code should satisfy the logic as checked by the tests.

Also, it would be great to see some examples in the README or in an examples folder. It's kind of hard for me to tell exactly what I need to write and what to call to generate the Ruby.

Sorry for the sparse documentation, I plan to improve that. I have added an example to the readme. Other examples can be found in `test/benchmark` folder.

smaudet3• 4 years ago

You still write the tests in standard Ruby and synthesized code should satisfy the logic as checked by the tests.

I think its important that the code still be user-modifiable:

Take this function, y = 2 * x.

Write a test, which proves for all real numbers, that y = 2 * x.

Can't do it, can you?

Tests can't fully specify code - this effort is doomed to be incomplete.

evolve2k• 4 years ago

It would be wonderful if you could embed a short video that explains the concept and shows some practical examples.

What it can do now. What are the limits. What’s planned next? Are questions we have as first time visitors.

bingo-bongo• 4 years ago

But doesn't this just mean that instead of writing your code and logic in Ruby you're writing it in RDL?

..and write a test for it..? :|

noneeeed• 4 years ago

Interesting. Reminds me of my time at university using something called Progol. You gave it positive and negative cases and it's algorithm would (very slowly) try and come up with Prolog code that fit it.

It was an interesting idea but in reality we spent so much time having to spoon-feed it more and more carefully constructed examples that it just seemed pointless.

Not to say that this will suffer from the same issue, but I do wonder if the benefit will really be significant enough to make it worth it.

sankha938• 4 years ago

Indeed, there is rich prior work on example based program synthesis. Often these use inductive logic programming or aided by SMT solvers if these examples can be lifted to solver level values (think integers or booleans, or data types made from those).

However, not all program values can be easily lifted to solvers. So the goal with RbSyn is to allow the programmer to write the same tests they would have written anyway to check their program correctness, without any extra fiddling. As tests subsume examples, we expect more program behavior can be specified than just using simple examples.

I do wonder if the benefit will really be significant enough to make it worth it.

I have often pondered on what makes a program synthesis tool useful. I expect writing code as art as much as it is science, and people like to express code in the way they model system behavior in their head. In that regard, I do not think program synthesis tools will enable you to automate away large parts of code, but I do think it will automate mundane parts of a program; like writing utility methods, or filling in a partial program when enough information can be gathered from surrounding context (such as the arguments you write for a substring function).

willlma• 4 years ago

I'd love to give this a try next time I'm working on a Ruby codebase. In the meantime, I'm curious to learn more about programming by example. I found [this Microsoft paper on the subject](https://www.microsoft.com/en-us/research/wp-content/uploads/...) and have started reading it, but I'm curious if there are any reading materials you would recommend.

Specifically, I've been working on a activity tracking app that can take plain text and create structured data from it. For example

I played tennis with Kevin and won 6-2

would turn into

```

type: tennis

opponent: Kevin

my_score: 6

their_score: 2

outcome: win

```

As you can imagine, when it comes to natural language there are many variations of I won (outcome: win), he/she/they won (outcome: loss), I was beat, etc which means that my inputs are not nearly as structured as the examples used in the paper I'm reading. Given that I'm not trying to cover every use case in the English language, which I assume would require some form of NLP, but rather just a subset that a single user would input, do you think think it would be possible to solve this with PBE techniques?

pmontra• 4 years ago

I'm writing a Rails API app (for a customer) that is a client to Alexa's Skill Management API and is a server for a backend app. I have tests but I'm afraid that without a 100% coverage of all corner cases I'll be disappointed by some results and it could be difficult to manually fix the code generated by the program. Am I overly pessimistic?

sankha938• 4 years ago

That is a fair concern. The tool only guarantees correctness up to the level checked by the tests you provide. So if all corner cases are not covered, RbSyn will generate some program that passes the tests, but might not pass unspecified corner cases. I suspect the missing cases will be clear with a manual audit of the synthesized code, and you can add update the synthesized code manually or add requisite tests.

giraffe_lady• 4 years ago

I suspect the missing cases will be clear with a manual audit of the synthesized code

We might just have really different brains but "read the code and figure out everything it doesn't do" is one of the hardest possible tasks for me?

Show HN: Program Synthesis for Ruby

Related Stories