A programmable programming language (2018) [pdf]

https://cs.brown.edu/~sk/Publications/Papers/Published/fffkbmt-programmable-prog-lang/paper.pdf

by azhenley • 7 years ago

68 47 7 years ago

cs.brown.edu

sam_lowry_• 7 years ago

LISP?

DannyB22• 7 years ago

Racket is a Lisp. One that facilitates building other languages.

schwurb2• 7 years ago

You could characterize Racket as THE lisp when it comes to well-behaved and well-researched metaprograming.

kleer001• 7 years ago

That was my first thought.

yters2• 7 years ago

I used Racket for a couple projects, but meta programming hasn't turned out to be super useful. When I did use meta programming, I got pretty confused when returning to my code after leaving it for awhile.

I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today.

I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.

skohan• 7 years ago

I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.

I've found Swift to be extremely good for this. It might be hair slower than python to get a project up and running in that very early stage, but it has a similar feeling of "writing the intent of the program not the syntax of the language" and the type system and compile time checking help alleviate a lot of the growing pains you might run into when a project starts getting medium-sized.

wry_discontent2• 7 years ago

I've found Python to inhibit me a lot from moving quickly past the very initial stages.

I'm fond of Clojure for that, but it tends to de-emphasize macros as a good way of doing things. Most things are just regular fns.

yters2• 7 years ago

Do you find it is easy to come back to your Clojure code and understand what is going on?

Ididntdothis2• 7 years ago

“I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case, and even if true whether that is still relevant today”

I can totally see this if you are alone or a very small tight team is working on the code. You will know what’s going on but once you start adding more people it will be very hard for them to work with such a codebase. I have done some really elegant stuff that made me super productive but looking at the code three years later I myself can barely figure out how it works or how to make changes. other people have almost no chance.

Once you grow it’s probably good to stick to paradigms a lot of people in the Industry are familiar with.

phlakaton• 7 years ago

I have done some really elegant stuff that made me super productive but looking at the code base three years later I myself can barely figure out how it works or how to make changes.

Could well apply to me and some of my old C++ projects. Well, except for the super-productive part.

PG would say that it's not that the abstractions afforded by metaprogramming are incomprehensible. It's just that, if you are a habitual Blub programmer, everything up the power curve from you is always going to appear weird and hairy.

This sort of argument is a terrible one to have with generalizations, though. You should look at specific cases where metaprogramming has been used and analyze them on a case-by-case basis. Sure, a bad, leaky, poorly-documented abstraction is going to lead to grief. On the other hand, umpteen thousands of programmers use the metamagic of Rails on a daily basis and barely bat an eyelash. (Or so it seems to me.)

jlarocco• 7 years ago

After years of using Common Lisp, my opinion on "meta-programming" is that it's wonderful when used sparingly and in just the right way, but a bad idea 99% of the time. If it's not completely obvious what (not how) a macro does just by looking at the code that calls it, then it's almost certainly a bad macro.

For example, there's a CL library, iterate, that introduces several new iteration constructs. There are a bunch of complicated macros under the hood, but as an end user I never have to think about them, and the provided interface is obvious, even without knowing anything about the iterate library. For example:

    (iterate (for (key . item) in alist)
      (collect key into keys)
      (collect item into items)
     (finally (return (values keys items))))

I would definitely consider, "Just because you can doesn't mean you should" to be the number one rule of metaprogramming.

ralphc• 7 years ago

I think of Lisp vs. Python similarly to PostScript vs PDF. In PostScript you have a full language, can do anything but you often have to do it yourself. PDF's aren't extensible but have _just about_ all the functionality you need to produce a document.

An example of macros you often see is surrounding a block of code with open/close pairs to make sure you don't exit the code leaving dangling resources. File open/close, database connections, etc. Python has the __enter__ and __exit__ protocols to handle that. You can't do every bit of metaprogramming in Python, but over the years they've covered close to every contingency a developer comes across.

lispm7• 7 years ago

Lisp uses advising for that. In the modern form these are :before, :after and :around methods in CLOS.

The difference between some Lisps and CPython is this: Lisp is its own implementation language and provides lower level control flow constructs, which can be compiled to efficient code.

pdonis2• 7 years ago

I know PG claims meta programming was the secret to ViaWeb's success, but I'm not sure how exactly that was the case

IIRC, PG said the advantage of Lisp was being able to add features much more quickly than the competition, so that (a) they could be adding features almost daily that the competition would take weeks or months to match, and (b) whenever the competition added a new feature that looked like a good idea, they could implement it within a day or two, so by the time the press came around to ask them if they had it too, they could say yes.

I also remember him saying that Lisp macros were about 25% of Viaweb's code, so a significant part of the advantage of Lisp was metaprogamming.

even if true whether that is still relevant today.

It's probably not that much of an advantage any more if your competition is other startups, since they probably have access to basically the same metaprogramming tools that you do.

It might still be a significant advantage if your competition is large, mature businesses whose business model you are trying to disrupt.

I have found Python to hit the sweet spot much better in terms of rapid prototyping, and providing a runway to making the prototype more rigorous and maintainable.

I have found this to be true as well. But I also find that I do a significant amount of metaprogramming in Python, and one of the key things to transitioning to more rigorous and maintainable code is to make sure that whatever metaprogramming needs to stay there for the long term is packaged into a library or a tool whose internals won't need to change.

alwalo• 7 years ago

Could you write something more on metaprogramming in Python ? And how does this compare with Lisp, what do you use it for ? Any good resources to learn this ?

agalunar• 7 years ago

One of my best friends did his Masters work designing a programming language that can be modified on the fly; I'd highly recommend the paper:

https://dspace.mit.edu/bitstream/handle/1721.1/113144/101799...

nielsbot• 7 years ago

The API for building grammars seems straightforward to use.

jimbob45• 7 years ago

I feel like there's something so comforting about having a concrete language you can rest your back up against though. Once you start throwing in #DEFINEs and typedefs, I start to question whether I can trust anything I'm seeing on the screen to not be hiding crucial secrets from me.

Maybe it's just me.

kd0amg• 7 years ago

You don't know what all those library calls do behind the scenes either.

bob1029• 7 years ago

Having constraints imposed upon me by my tools is the only way anything ever gets done. If you put me in front of a blank canvas, and then tell me I can alter the nature of the canvas itself to whatever extent I desire (I.e. in hopes of making using it easier), I'll never get to the point where I'm actually painting on it. Or, perhaps I do manage to produce an end result, but no one else can understand what it means.

Even with a mountain of constraints, there is still usually far too much subjectivity in software development. This language seems to me to be moving the needle in a very wrong direction. Having a stable basis that is standardized and everyone can (mostly) agree upon is the foundation for growing a technical team or community. Being able to type "LINQ GroupBy Select examples" into your preferred search engine of choice and receive actionable advice is clearly an advantage for productivity. Sure, in a purely academic sense there is almost certainly a more ideal solution. But, if you are seeking to actually get things done, constraints are the name of the game.

discreteevent• 7 years ago

I remember some interview with Ulrich Schnauss (electronic artist) where he went back to plain synthesisers for the same reason. Too many things to tweak meant he wasn't able to focus on the piece.

zomglings• 7 years ago

What you said really resonates with me - especially re: resting your back up against it.

Just built my first single-page website with React and I ran up against the new Javascript ES6 import system. It makes no sense that people keep introducing this kind of unnecessary complexity into their tools. What was wrong with good old "require"?

This is just one instance among many many many. It feels like I'm aging a year every day.

DannyB22• 7 years ago

Once you start throwing in #DEFINEs and typedefs,
I start to question whether I can trust anything I'm
seeing on the screen to not be hiding crucial secrets from me.

What could go wrong? Try my cool optimizations!

#define while if // saves cpu cycles

#define struct union // saves memory

schwurb2• 7 years ago

#define while if // saves cpu cycles

lol, thanks for the laugh!

AtlasBarfed• 7 years ago

No, this is precisely how most Lisps shoot themselves in the foot. When you can do anything, people make these walled gardens/ivory towers of processing frameworks and libraries.

The users of Lisps are often very intelligent single-hero developers as well (why, Paul Graham's Lisp article is a perfect example of it).

And since documentation is ALWAYS terrible for anything except trivial things...

dang• 7 years ago

Lisps don't shoot themselves in the foot any more than chainsaws cut their own limbs off.

Koshkin• 7 years ago

The usefulness of this depends on how you want to solve the problem at hand. One approach to programming is to create a DSL for the problem domain and then solve the problem using that DSL. Languages (such as Common Lisp) that include advanced means of metaprogramming are perfect from this perspective. Otherwise, one has to create an API to approximate the DSL - which could be rather hairy, so one may try to manage the complexity of this particular representation by either using a preprocessor or even, as an extreme measure, by writing a separate language processor (compiler or interpreter) for the DSL.

jmull• 7 years ago

I feel like general purpose programming languages very widely already let you express DSLs, with semantics, names, structure, etc. appropriate to the domain.

What they do restrict, to various degrees, is the syntax.

So what we're talking about here is custom/customizable syntax to go with your DSL. That can be nice, but it comes at a high cost -- e.g., developers will have to understand the syntax, it will need to be maintained and extended as requirements grow or change. Of course it's worth it in some cases, and tools to help do it are great when it is.

But I just don't see that there's a general problem that we don't have enough custom language syntax to deal with.

bitwize2• 7 years ago

Meh, in practice most people will use JavaScript and Babel.

iLemming6• 7 years ago

Yeah, and that is a problem with the majority of programmers. They readily choose a seemingly easy path instead of something that leads to simplicity. Lisp is a fantastic idea that (even after over six decades) is still very much relevant today. Once you get used to program with a true REPL, every other (non-lispy) language starts feeling like frustratingly over-engineered, bloated, and unnecessarily complicated. Waiting for compilation cycles and annoying syntax quirks would slowly turn what once was a joy into a "just a job."

It feels incredibly liberating and joyous to be able to grab any part of your program and without any preceding ceremony just to "eval" it and see what happens.

And let me prematurely answer the comments that probably will be thrown here. No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells".

Younger folks who are learning, juniors, seasoned developers who for one or another reason, never had any exposure to a Lisp, please do yourselves a favor. Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it.

Follow the work of people like Harold Abelson and Gerald Sussman, Matthias Felleisen, Dan Friedman, Rich Hickey, Guy Steele, Richard Gabriel, Alan Kay, and many others.

Once you attain familiarity with a Lisp to the sufficient level, perhaps then you will get it. You will see that it is indeed a lifetime investment that would allow you to reap the rewards for the entirety of your career.

I am telling you all this, not from the top of some ivory tower. I never went to CS academia. Like most of you, I am altogether a self-taught programmer. I am honestly begging you not to make the same mistake that I did. For many years I have ignored Lisp due to my self-righteous ignorance. Unknowingly, I delayed my true growth, and I sincerely regret it.

lispm7• 7 years ago

No, other languages (most of them) don't have real REPLs. What they have, at best can be categorized as "interactive shells".
Try Racket, Clojure, Clojurescript, Fennel, Chez. Give Lisp an honest, heartfelt attempt to learn it.

Compared to actual Lisps, some of these have mostly interactive shells.

https://www.youtube.com/watch?v=o4-YnLpLgtk

bitwize2• 7 years ago

Most Lisp programmers go through hype cycles within their own minds, progress along which increases with experience. You appear to be still at the Peak of Inflated Expectations phase of your personal hype cycle. You've learned enough Lisp to be dangerous, and you've seen, and been tempted by, its tremendous power. Oh, the things you could build, if only you were allowed! But be forewarned, the Trough of Disillusionment is coming, the despair that sets in when you realize that try as you might, you cannot convince industry to accept its software needs being met in Lisp. For as mighty as Lisp is on its own compared to other languages, other languages have surpassed it in the ways that matter to companies: better integration, better tooling, more support, bigger talent pools -- and lower cost for all of these.

Lisp programmers tend to spend their Slope of Enlightenment and Plateau of Productivity working in other languages, applying what they learned from Lisp to Python or Ruby or Haskell or Rust -- or JavaScript. And honestly, JavaScript with Babel gets you 80% of the dynamism and metaprogramming capability Lisp has, but also better integration, tools, and libraries than any Lisp. So yeah, most people in the real world will use that.

A programmable programming language (2018) [pdf]

Related Stories