The Nature of Lisp (2006)

http://www.defmacro.org/ramblings/lisp.html

by taheris • 11 years ago

110 67 11 years ago

Slackwise• 11 years ago

This was the page that made me finally grasp why Lisp and s-expressions are relevant. After reading this about a year ago, I ended up picking up a Clojure book and now I'm enamored with s-expressions. I feel late to the party.

The last time I tried to learn Lisp (prior to reading this) was in like 2003/4 from some CLISP tutorial that scared me away from having nothing but math examples and zero explanation of the philosophy or design. It felt like an esoteric and purely academic language, and I brushed it off as not worth my time. (I was never really put off by the parens, and now I love them. Never understood that aspect of Lisp discourse. Once you learn ParEdit[1] you don't want to go back.)

Now I link this page whenever I want to explain to someone why there are "all these parens". It's dated, though, and associating s-expressions with JSON and JavaScript would probably be a more contemporary approach. Maybe I should rewrite this article for a modern developer audience.

[1] http://danmidwood.com/content/2014/11/21/animated-paredit.ht... "GIFs of ParEdit in action"

frou_dh• 11 years ago

There's a "lisp parens lol" thing among programmers that's so intellectually lazy. I once interviewed someone who I noticed had explicitly mentioned Lisp on their resume, and when as an aside I asked what their thoughts on Lisp development were, all they could muster was "Oh god the parens hyuck hyuck".

mbrock7• 11 years ago

Other languages tend to get a free pass, but I take every chance I get to whine about JavaScript's three kinds of parens that I have to keep track of manually, and the convention of putting nearly every closing paren on its own line, wasting kilometers of space.

agumonkey6• 11 years ago

The mainstream may realize how syntax doesn't really matter. The FP push helps a lot already (map,filter,streamfusion instead of crazy for/while statements).

aaron-lebo5• 11 years ago

I don't know that it is intellectually lazy. It makes a lot of sense.

If a lisp dialect had a large, vibrant ecosystem on the scale of Python or Javascript, then a lot of people would see the parens, see the advantage of having them, work their way through it and eventually get used to them and even embrace them because they see a real payoff.

That's not the case with lisp dialects. The major payoff with lisp dialects is macros. But for most people, it isn't obvious why they are important. They are something you have to use to really get.

So the intellectually reasonable position for most people is to either ignore lisp or laugh at it. Meanwhile, you see endless articles like this posted over the years (not knocking the article itself), trying to explain lisp in a kind of reverence and terminology usually associated with religious texts. "Hey, brother, you just need to see the light".

I've been doing quite a bit of Clojure coding lately and I love it. I just can't see a lisp ever catching on. Actually, the project I'm working on is a lisp that does not look like a lisp but compiles down to Clojure. The parens aren't really necessary for lisp's best feature, as seen by Elixir. Here's a bit of my language:

    " " join([1, 2, 3]) count println

j_m_b• 11 years ago

I really like the Clojure's minimalistic EDN format syntax. In comparison, Scheme and Common Lisp seem like they have superfluous parens.

breuleux2• 11 years ago

I think most people have an irrational dislike of s-expressions, but that at the same time, s-expressions are not as good as their proponents think they are. There's a certain aesthetic "purity" to them that's seductive, but from a practical standpoint, it is quite reasonable to adapt syntax to common patterns. The issue with mainstream languages isn't that they do that, it's that they do it too much.

But there are syntactic schemes besides s-expressions that allow for good flexibility. For instance, in my programming language Earl Grey[1], I use this very simple sugar: `a b: c` <=> `a(b, c)`. So you can use this todo syntax if you want, and it will just work:

    todo "housework":
       item priority(high): "Clean the house."
       item priority(medium): "Wash the dishes."
       item priority(medium): "Buy more soap."

Interestingly this has one advantage over s-expressions: whereas Lisp editors usually need to specify a special highlighting/indent policy for keywords like `let`, here editors can just highlight/indent `a b: c` indiscriminately.

I also think a limited number of infix operators are worth defining in most languages, and I don't mean arithmetic operators (although I'm okay with them). I mean ubiquitous or easily repurposed operators like assignment/declaration, lambda, pairing, or commas. In my view, Lisp-like languages hold to standards of purity that are unreasonable (don't get me wrong, they are quite usable, just not optimal according to any useful metric).

[1] http://breuleux.github.io/earl-grey/

drakecoffeecake• 11 years ago

I don't think that's much clearer than a lisp equivalent:

    (todo "housework"
        (item (priority high) "Clean the house.")
        (item (priority medium) "Wash the dishes.")
        (item (priority medium) "Buy more soap."))

Yours looks nicer, but if I wanted to operate on your todo there, it's not obvious how just by looking.

With the lisp, it's obvious: There's a list of 5 elements. The last three are 'items', each of which is a list of 3 elements.

The benefit of s-expressions isn't just that they're flexible in what you can write with them, but that they're easy to manipulate programmatically, and their structure is immediately obvious.

mbrock7• 11 years ago

For reference, an S-expression based syntax that could be used for this example:

    (todo "housework"
      (item :priority high "Clean the house.")
      (item :priority medium "Wash the dishes.")
      (item :priority medium "Buy more soap"))

Your syntax certainly has advantages, but one can easily prefer S-expressions in this case without appealing to vague values like "aesthetic purity," nor are there any blatantly obvious "practical" downsides to using a more fully parenthesized syntax.

auganov• 11 years ago

Mathematica is a pretty good example of a language that introduces some syntax without compromising on homoiconicity.

For example

  a := {1 + 2 - 10, 10, {1, 2}}

is just

  Set[a, List[Plus[1, 2, -10], 10, List[1, 2]]]

brandmeyer3• 11 years ago

One of the aspects of the data-is-code and code-is-data duality that always bugged me surrounds security. Ever since we started embedded macro languages into documents, there have been classes of security problems around them. Even before the web there were macro viruses in abundance, and running spreadsheets with macros embedded in them is still a dangerous proposition today.

How does Lisp deal with the modern Writable Xor Executable philosophy? Or does it? Can a lispy program that exposes its lispyness to its users be secure (emacs, I'm looking at you)?

agumonkey6• 11 years ago

AFAIUI you write restricted evaluators to deal with external code. You can dismiss or rewrite any form you want, avoid IO, system calls and such. I never looked into it so maybe it's possible to escape that, it's been used on an irc bot for #clojure, so it's not stupid; but there's always a possibility for crazy ideas like crafting memory patterns and trying to DDoS the evaluator in order for these to become code (complete speculation based on Return Oriented Programming).

brandmeyer3• 11 years ago

Combining syntactic closures with the Scheme module system's ability to carefully control exports and imports seems like it should be enough.

However, high-security language runtimes have a history of being subverted anyway.

raphaelss• 11 years ago

Just like every other language.

You either

- Expose the entire language and then there's not much you can do to prevent something harmful.

- Expose a subset of the language perhaps through parsing for harmful patterns or by somehow making potentially harmful functions unreachable.

- Expose a more restricted language (scripting like c/lua) that you control and know its full capabilities.

The only difference I can think of is that it's easier to write flexible and dynamically modifiable software in lisp than in most mainstream languages.

seiji2• 11 years ago

The end result has been years of (practical) research going into an actually safe, actually sandboxed, actually secure cross-platform platform: https://developer.chrome.com/native-client

btilly• 11 years ago

Capability based systems offer better options.

All access to anything important (network, files, other commands, etc) have to happen through opaque handles called capabilities. Think of them as objects that you can't look for.

When you call code, you pass it a set of capabilities that are available. That is all it can ever access.

But the whole language and environment needs to be defined from the bottom up to enable this.

mbrock7• 11 years ago

Generally, you just don't expose your Lisp evaluator to untrusted users. Lisp macros are a powerful language feature, and to allow your user to evaluate arbitrary code is exactly as dangerous as with any other system.

Emacs is not meant to be secure against user scripts. You could even say that's an integral part of its whole philosophy: the user can do whatever she likes, and her scripts too.

It does have some rudimentary safety. For example, color themes by default can't run Lisp code. They are written as Lisp data structures, but they aren't run through the evaluator unless the user explicitly allows it.

kyllo• 11 years ago

It's up to the programmer to decide what code/data is trusted and untrusted. So you wouldn't `eval` any S-expressions that could be submitted by a user. It's really no different in terms of security implications than any other dynamic language like Ruby, Python, Perl, PHP, Javascript...

acc543212• 11 years ago

Don't use read or eval willy nilly. If you have to use read, at the very least turn off read-eval. I give up on trying to get an asterisk on each side of that variable. No form of escaping seems viable.

sklogic3• 11 years ago

What does code-is-a-data have to do with security?!? Do not confuse an exposed `eval` with a compile-time homoiconicity.

AnimalMuppet4• 11 years ago

Lisp syntax looks really good after you try doing the same thing in XML...

spacemanmatt2• 11 years ago

...shades of Spring application.xml

ExpiredLink• 11 years ago

Lisp is a way to escape mediocrity and to get ahead of the pack. Learning Lisp means you can get a better job today, because you can impress any reasonably intelligent interviewer with fresh insight into most aspects of software engineering. It also means you're likely to get fired tomorrow because everyone is tired of you constantly mentioning how much better the company could be doing if only its software was written in Lisp. Is it worth the effort? Everyone who has ever learned Lisp says yes. The choice, of course, remains yours.

roneesh• 11 years ago

I've read and heard the commonly echoed sentiments about Macros being used to build a language for your application. I'm a bit confused in how this is philosophically different than just writing domain specific classes and functions? Can someone point to an example where the syntax allowed by Ruby or JS produced obfuscated code and a LISP macro produced something really simple comparatively?

AnimalMuppet4• 11 years ago

It's the difference between introducing new nouns and verbs, and introducing new grammar.

With new classes, I add new nouns (Circle, for example). With functions, I add new verbs (move, for example). And I can add adjectives and adverbs (attributes and arguments). This lets me build a "new language" (for handling shapes, in my example).

But macros let you go much further. They let you change the syntax, so that instead of saying "shape.move(newCoords)", you can say "move shape newCoords", for example, even though that was not legal syntax in the language you're writing in. That's where it really becomes a new language - not just new words.

But if you've got "good enough" syntax in the language you're writing in, you might not see that as particularly useful. That is, the less impoverished your out-of-the-box implementation language syntax is, the less you need macros. (Most of the examples say "it's awkward to write this; I can write a macro to make it easier". Well, that's great and all, but awkwardness of writing something that I don't write very often in the first place does not make me want to run out and adopt a macro language.)

malisper• 11 years ago

My personal favorite is the series library. I wrote a bit about it here: https://news.ycombinator.com/item?id=9078155

sklogic3• 11 years ago

Statically compiled regular expressions. Efficient pattern matching implementations. Generating a boilerplate for an ORM. Embedding languages with arbitrarily complex syntax. There are hundreds of things that you simply cannot do with functions but are trivial with macros.

TheMagicHorsey• 11 years ago

With macros being so easy to write, and so idiomatic to the way the language is used, isn't it very difficult to understand a Lisp codebase that you are unfamiliar with?

While I like Lisp a lot for my own projects (well Racket in Dr. Racket to be specific), I don't feel very comfortable trying to understand other people's code.

Other languages like Go, for example, seem a lot more tractable when you have to grok a large foreign codebase.

sklogic3• 11 years ago

If macros are used properly, i.e., for implementing nice, multi-staged, composable eDSLs, instead of just another anaphoric if or similarly useless "extension", then a Lisp codebase is much easier to read than anything that does not allow implementing macro-based eDSLs.

mbrock7• 11 years ago

Do you have concrete examples of this difficulty?

I very often find it difficult to understand code in non-Lisp languages, too. For example, you may think you know Ruby, but that has nothing to do with understanding Rails or its myriad metalinguistic tricks.

Abstractions are often difficult to understand, and require explanations. Introducing a new abstraction, regardless of implementation language, puts the burden on you to define it with clarity, to document its purpose and meaning, and so on.

Lisp macros let you define abstractions in a certain way that's quite powerful. Fortunately, Lisp systems also tend to have good provisions for online documentation, and make available macro-expansion facilities that immediately explain exactly what a given macro is doing.

dllthomas• 11 years ago

In general, the more you can package up into single abstractions, the more someone will have to learn to thoroughly understand your code base. If those abstractions fit the problem domain well, hopefully they will ease the overall cognitive load of working in the system, even somewhat early on, but there will always be some ramp-up time. I'm not at all sure it's a big problem, just something you need to be aware of.

soegaard• 11 years ago

Experience says no.

There are several reasons, but

    * macros are only used when functions doesn't make sense
    * there are documentation
    * most follow the conventions of the language

chinpokomon2• 11 years ago

I started trying to learn FP this last year. I started trying to understand Clojure and then watched the SCIP lectures and worked through the problems in the book with the class. At this point, I think I _get_ LISP.

What I don't get is how I build an application using LISP. From the article, the todo list wasn't really a list, it was a todo function call that took tasks as arguments unless there was a missing quote. That seems to be a point missed by the author. With the macro, he was able to format the tasks themselves, but after applying the macro you'd still be left with a todo function and done arguments. I don't see where that gets called.

LISP seems to work best as an evolving environment. I can see where a LISP machine could be powerful for defining your own DSL as you try to parse through some data. I also saw the power in how you abstract the problem, really taking advantage of code reuse in a powerful way. I don't see how you write an application.

Is there some resource or there that can help me bridge this gap? I feel like this is like trying to use Mathematica to write a program. Mathematica is great for calculating as LISP seems great for processing data, but I'm stuck in seeing how I can utilize it for anything beyond a contained process. At least with Clojure or F#, I could still use Java or C# for the entry point and call into an FP block of code for calculations and processing. How would I do something with Haskell or Scheme?

karamazov• 11 years ago

Eventually, some of your code will make a call to IO: reading from or writing to disk, responding to a web query, updating a DB, drawing something on the screen, etc, etc.

For example, you could put up a web server written entirely in Scheme or some other dialect. This would process and respond to queries entirely in Scheme, including some sort of engine for generating and serving html. Hacker News itself is an example of this: it's written in a lisp dialect called Arc.

Incidentally, Haskell is functional, but it isn't a lisp.

chinpokomon2• 11 years ago

Yeah, I know that about Haskell, but my question was about FP in general. I didn't know that about HN. I'll just have to keep looking I guess.

The Nature of Lisp (2006)

Related Stories