Why Lisp?

http://blog.rongarret.info/2015/05/why-lisp.html

by lisper • 11 years ago

322 239 11 years ago

blog.rongarret.info

rdtsc2• 11 years ago

The act of describing what you want the machine to do is interleaved with the machine actually doing what you have described, observing the results, and then changing the description of what you want the machine to do based on those observations.

That sounds really powerful and interesting, but wondering how often is that used in practice? I can imagine maintaining and understanding a large self-modifying program like that could get a bit complicated for a team.

gcv• 11 years ago

I think Ron was referring to the development process, rather than the finished program. The way Lisp languages and environments handle REPL-based interactive development is significantly more powerful and flexible than e.g., Rubython or even Haskell.

Modifying a running program to make a change as you develop it is an incredible speedup, relative to (recompiling and) restarting all the time, and even relative to reloading whole changed files.

orthecreedence8• 11 years ago

I think he's describing interactive programming, not necessarily self-modifying programs. With that said, yes interactive programming is incredibly useful and I do it all the time.

It really shines on apps with a lot of state (such as a game) where normally you'd have to quit, change the code, recompile, run the app, and reproduce the original state as closely as possible. With lisp you can just replace the function you are debugging while the app is running.

That said, lisp can be used for self-modifying programs pretty easily. The idea that the default data structures it manipulates are the same as the code itself makes it easy to build data structures with the purpose of being evaled (not saying this is ever a good practice).

And essentially, macros are self-modifying programs that run before the actual program runs.

rdtsc2• 11 years ago

I think he's describing interactive programming, not necessarily self-modifying programs. With that said, yes interactive programming is incredibly useful and I do it all the time

I do it too in python via ipython. Maybe what the author meant is "Why exploratory programming using a REPL is great".

jeremiep• 11 years ago

The powerful idea is the ability to modify the program as it is running rather than writing a self-modifying program which indeed would be much more complicated.

A good example would be the Emacs text editor. You can use it to modify the source of Emacs itself while it is running. You never need to reboot it to test new code because you can interactively evaluate new functions and replace existing ones.

As for working in teams, usually each developer runs their own REPL instance isolating them from the changes made by other developers. You can still pull a coworker's commit and evaluate it into your running instance without even restarting the program under development.

lisper7• 11 years ago

Like any powerful technology, Lisp's interactivity and dynamism can be used for good or it can be used for evil. Yes, it takes a little discipline to keep things from spinning wildly out of control. But it's well worth the effort.

enupten• 11 years ago

Precisely. I find that getting the balance (that ethereal "style") right takes a lot more time than actually "learning" the language.

orthecreedence8• 11 years ago

The interactive model is insanely cool. When building a toy game engine a while back (https://github.com/orthecreedence/ghostie) I saved probably half the development time by being able to redefine functions/values while the game was running.

The old way of lisping is to prototype in lisp, then build in a "real" language (c/java). However nowadays the lisp implementations (CCL/SBCL specifically) are fast/advanced enough to the point where you can prototype in lisp, then just use add some type specifiers and boom there's your app. Hell, with ECL you can even embed your lisp program into another one, while still achieving compiled speeds.

demilicious• 11 years ago

What sort of tools are required for this? I'd like to start taking Lisp seriously, but workflow stories like this tend to hinge on using Emacs/SLIME. Is that always the case?

EDIT: I suppose what I'm asking is whether you could elaborate more on what this looks like, in practice.

orthecreedence8• 11 years ago

Actually no, I use vim/slimv almost exclusively. I think Sublime2 might also have features that let you "hook into" a remote REPL, but I'm not very familiar. There may be other editors with lisp integration as well, but I'm not sure what they are.

If you do like vim, slimv is a really great option for lisping.

Jach2• 11 years ago

I use vim with "vim-slime", which is different than "slimv". https://github.com/jpalardy/vim-slime

Basically I have a horizontally split 'screen' session, code in the top and repl in the bottom, and I just ctrl+[c+c] to send paragraphs from the top to the bottom. I don't remember if vim-slime comes with it or if I augmented it, but I also do ctrl+[c+f] to send the current Lisp/Clojure form, ctrl+[c+l] to send the current line... Or just ctrl+a+tab to switch screen windows and type in the REPL directly.

stonemetal• 11 years ago

Allegro Common lisp comes with an IDE with all the bells and whistles.

weavie3• 11 years ago

Even better you can attach the repl to a remote instance. I had a problem a little while ago that could only be reproduced on the server. I could connect to the repl over ssh and evaluate and modify code directly.

Compare that to a similar problem I had with a C# app we had. For that I had to stick in a load of logging code, check it in then wait half an hour for the CI server to deploy before running and checking the logs.

Working with a Lisp was much more pleasurable.

dmux• 11 years ago

Do you know if the attaching to a remote instance is available in Racket? I spent some time learning racket a year or two ago, but was under the impression they took out some of the really cool features (or I never discovered them)

sirclueless2• 11 years ago

You can get this in other languages too. For example, Flask (a Python web framework) has a fantastic debug-mode error page that totally changed the way that I think about web development. Any time an exception is thrown in a view function (and this includes the exceptions that you idiomatically throw for HTTP 4xx and 5xx errors) the debug-mode error page would have a stack trace (obviously), but also an interactive REPL that could be opened at any stack frame in that trace. It wasn't necessary all that often, but when it was, boy was it a fantastic way to work.

icosahedron• 11 years ago

Which lisp do you use? I've used SBCL and tried using mocl and ECL on mobile platforms, but without any real luck. Too much work getting them set up.

orthecreedence8• 11 years ago

Clozure CL is easy to install/run on just about any desktop platform (windows included). On *nix, SBCL is a good choice.

I've heard good things about MOCL but haven't tried it myself, and I know getting ECL working on mobile is an uphill battle (but achievable if you have the time). ECL switched maintainers recently, so maybe mobile is something they will focus on in the future.

WalterGR6• 11 years ago

Seconding the Clozure CL recommendation. It supports threads on Windows, which really expands your options if you want to do web development in Common Lisp.

I run Linux on my web servers but develop from a Windows desktop. CCL works great on both.

nocman• 11 years ago

The only downside to Clozure CL that I found was the fact that it requires SSE2 instruction support from the processor. There are still some processors around that don't support that -- which can be a bummer if you want to use Clozure CL on one of those machines. Unfortunately, one doesn't always have the option of upgrading the hardware to get around that.

I don't think the developers have worked around that, though I have seen some conversation about it in the past. My limited Google searches on the subject today didn't suggest that anything has changed since the last time I looked at it.

Edit:

Oh, duh, I already said all of this about 4 1/2 years ago:

https://news.ycombinator.com/item?id=1804415

Visiting the system requirements page again suggests the limitation still exists:

http://trac.clozure.com/ccl/wiki/SystemRequirements

wukix• 11 years ago

Did you reach out to me when you were using mocl? Surprised to hear this. AFAIK most users get up and running w/o much trouble.

kazinator15• 11 years ago

I'm hacking on TXR these days which contains a Lisp dialect called TXR Lisp. Lisp hacking and research is fun, in particular if you have the freedom of your own dialect.

The current public release TXR Lisp still has an embarrassingly shoddy implementation of "places": expressions which not only evaluate, but serve as assignable locations. I implemented most of the place-manipulating operators (set, inc, push, ...) as special forms, and only a small repertoire of places is hard-coded in the interpreter. (And that, by the way, creates one of the impediments against going to a compiler.)

I have thrown all that out and created a macro-based system for places. Instead of copying the Common Lisp one, I drummed up something new.

Common Lisp lets programmers register assignment places as "setf expanders" which return five values: lists of temporary variables, access and store forms and such. (Google for "CLHS get-setf-expansion").

I have taken a somewhat different, though necessarily closely-related approach. The creator of a new place provides two or three functions. These functions take a place and a piece of code, and wrap it with macrolets for accessing the place. These macrolets are given names that the caller specifies. Of course, I have a macro for writing these functions as a capsule.

The three functions provide a way for handling the access and update of a place, a simple store to a place without an access, and the deletion of a place.

This third item is new. In TXR Lisp, there is a del operator: a place can be vaporized. Not all places support deletion, but many do. For instance you can do (del [some-vector 3]) to delete an item from a vector. Or a range. Hashes support deletion. (del (car x)) works: what happens when you delete the car of a cons cell is that the cdr is popped, and the popped item is transferred into the car. Thus if x holds (1 2 3), the (2 3) part is popped to produce (3) and the 2 moves into the car, resulting in X holding (2 3). (I just realized that we need place insertion to complement place deletion!)

Here is how a (vecref <vector> <index>) place is defined:

   (defplace (vecref vector index :whole args) body
     (getter setter
       (with-gensyms (vec-sym ind-sym)
         ^(rlet ((,vec-sym ,vector)
                 (,ind-sym ,index))
            (macrolet ((,getter () ^(vecref ,',vec-sym ,',ind-sym))
                       (,setter (val) ^(refset ,',vec-sym ,',ind-sym ,val)))
               ,*body))))
     (ssetter
       ^(macrolet ((,ssetter (val) ^(refset ,*',args ,val)))
          ,*body))
     (deleter
       (with-gensyms (vec-sym ind-sym)
         ^(rlet ((,vec-sym ,vector)
                 (,ind-sym ,index))
            (macrolet ((,deleter ()
                          ^(prog1 (vecref ,',vec-sym ,',ind-sym)
                                  (replace-vec ,',vec-sym nil
                                               ,',ind-sym (succ ,',ind-sym)))))
              ,*body)))))

[[ Digression: what is rlet in the above?

The rlet macro is something I just invented days ago. It is like let, but it detects and optimizes cases when a symbol is bound to another symbol, and turns these into symbol macrolets. Demo:

  $ ./txr -p "(sys:expand '(rlet ((a b) (c (form))) (list a b c)))"
  (let ((c (form))) (list b b c))

Look, the (a b) disappeared in the form expansion, and b was replaced by a. That's because the macro expansion is this:

  $ ./txr -p "(macroexpand '(rlet ((a b) (c (form))) (list a b c)))"
  (symacrolet ((a b)) (let ((c (form))) (list a b c)))

rlet also propagates simple constants, as shown by the variable d:

  $ ./txr -p "(sys:expand '(rlet ((a b) (c (form)) (d 42)) (list a b c d)))"
  (let ((c (form))) (list b b c 42))

Of course, let can only be replaced by rlet in certain circumstances, when we don't rely on the binding actually providing storage semantics. Above, we can only replace a with b, if a is not assigned anywhere. And since d is replaced with 42 blindly, it cannot be assigned.

Obviously, rlet is something you don't need if you have an optimizing compiler which eliminates useless temporary variables and propagates constants! I just wanted nicer expansions from the places system.]]

So anyway, the defplace creates three functions which provide advice for how to correctly access, update and delete a place. They wrap this advice, which takes the form of local functions or macros, and any additional lexical definitions they need, around a supplied piece of code, and that code can refer to that.

Here is how this advice is used, in the implementation of the swap operator. This shows you the real beauty of the system, because we can write a completely robust swap, but we don't have to use anything resembling CL's get-setf-expansion and its five cumbersome return values:

  (defmacro swap (place-0 place-1 :env env)
    (with-gensyms (tmp)
      (with-update-expander (getter-0 setter-0) place-0 env
        (with-update-expander (getter-1 setter-1) place-1 env
          ^(let ((,tmp (,getter-0)))
             (,setter-0 (,getter-1))
             (,setter-1 ,tmp))))))

What is the env doing there? We have to pass the macro-time environment down, because places can be macros! For instance what if we do (swap a b), but a is actually a lexical symbol macro for (car x) and b is something similar: the with-update-expander macro will take care of expanding these places, so it fetches the correct advice for the expanded places.

So note how we have a completely naive looking pieces of code, a three point swap:

          ^(let ((,tmp (,getter-0)))
             (,setter-0 (,getter-1))
             (,setter-1 ,tmp))))))

All we did was generate some gensyms, use a couple of macros to obtain the place-accessing expertise from some generated lexical helpers, and then wrote a straightforward swap.

  $ ./txr -p "(sys:expand '(swap a b))"
  (let ((#:g0044 a)) (sys:setq a b) (sys:setq b #:g0044))

  $ ./txr -p "(sys:expand '(swap (car c) (cdr c)))"
  (let ((#:g0044 (car c))) (sys:rplaca c (cdr c)) (sys:rplacd c #:g0044))

  $ ./txr -p "(sys:expand '(swap 1 2))"
  ./txr: unhandled exception of type eval-error:
  ./txr: during expansion at string:1 of form (swap 1 2)
  ./txr: message: form 1 is not syntax denoting an assignable place

  $ ./txr -p "(sys:expand '(swap [f x] [g y]))"
  (let ((#:g0067 [f x])) (let ((#:g0056 [g y])) (let ((#:g0044 #:g0067))
  (sys:setq f (sys:dwim-set f x #:g0056)) (sys:setq g (sys:dwim-set g y #:g0044)) #:g0044)))

Note the [] notation is completely generic, so f and x could be lists (or vectors, hashes, strings). In this case, f and x are themselves expected to be places. Thus [f x] must be a place, and f also must be a place. The reason is that if we manipulate a list, we may have to assign the new list to the old variable. In general, sys:dwim-set could cons up a new list.

So in summary, Lisp is always fresh and keeps me interested in hacking. Even old things (solved problems like assignment places) can be looked at in a new light.

Check out this recursive implementation of shift, which is like Common Lisp's shiftf:

  (defmacro shift (:env env . places)
    (tree-case places
      (() (eval-err "shift: need at least two arguments"))
      ((place) (eval-err "shift: need at least two arguments"))
      ((place newvalue)
       (with-update-expander (getter setter) place env
          ^(prog1 (,getter) (,setter ,newvalue))))
      ((place . others)
        (with-update-expander (getter setter) place env
          ^(prog1 (,getter) (,setter (shift ,*others)))))))

  $ ./txr -p "(sys:expand '(shift a b c d))"
  (prog1 a (sys:setq a (prog1 b (sys:setq b (prog1 c (sys:setq c d))))))

teddyh• 11 years ago

The rlet macro [in the above code] is something I just invented days ago.

This perfectly illustrates both the power and danger of Lisp; also known as “The Lisp Curse”¹

① http://www.winestockwebdesign.com/Essays/Lisp_Curse.html

Guvante• 11 years ago

The expressive power of Lisp has drawbacks. There is no such thing as a free lunch.

This line doesn't hold on its own but as the ending of the article it was very powerful.

I had heard that extra power can make it easier to do things wrong, but the idea that extra power can cause problems even when you do things right is fascinating.

agumonkey2• 11 years ago

WalterGR6• 11 years ago

The article doesn't discuss macros, which is one of the answers to "Why Lisp?"

I didn't "get" macros until I read a footnote in the (freely available) book Practical Common Lisp. In chapter 7, it introduces the `dolist` macro.

    DOLIST loops across the items of a list, executing the loop
    body with a variable holding the successive items of the list.
    This is the basic skeleton (leaving out some of the more
    esoteric options):

    (dolist (var list-form)
      body-form*)

    ...For instance:

    CL-USER> (dolist (x '(1 2 3))
               (print x))
    1
    2
    3
    NIL

Buried in a footnote is this:

"DOLIST is similar to Perl's `foreach` or Python's `for`. Java added a similar kind of loop construct with the 'enhanced' for loop in Java 1.5, as part of JSR-201.

Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide "expert group" to hash everything out. That process--according to Sun--takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer's favorite compiler supports the new version of Java, they probably still can't use the new feature until they're allowed to break source compatibility with older versions of Java.

So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years."

http://www.gigamonkeys.com/book/macros-standard-control-cons...

lisper7• 11 years ago

And if you want Python-style iterators, that's easy too:

http://blog.rongarret.info/2008/02/joy-of-iterators.html

ScottBurson5• 11 years ago

So, that thing I mentioned yesterday that I wrote 35 years ago? It might interest you:

  https://github.com/slburson/misc-extensions

Look in src/gmap.lisp.

I admit it wouldn't work that well for 'n-at-a-time', but here's a closely related example:

  (defun alist-to-plist (alist)
    (gmap :plist nil (:alist alist)))

There are some idiosyncrasies here: 'nil' is shorthand for the identity function (in this case, the two-argument, two-value identity function), and the use of keyword symbols is weird and potentially problematic, though I've never run into a problem with it in practice.

This is the map-reduce model of iteration -- obviously not with support for distributed computation, but nonetheless useful for programming in the small. And far more elegant, to my eyes, than that LOOP monstrosity :-)

IshKebab• 11 years ago

On the other hand, this really hurts readability. When reading other people's code you now effectively have to learn what "language" they use too.

I'd say it's probably worth that cost, if used judiciously.

kazinator15• 11 years ago

"If used judiciously" is just FUD.

Soviet-era authority figure: "Western-style freedom has its good points---if applied judiciously".

If you're a proper Lisper, you use macros like it's going out of style, and other proper Lispers love your code for it.

All programs have their own dictionary of whatever it is they define, whether it be macros or variables. You can no more understand a function call just by looking at it than a macro call. (Should functional decomposition be introduced "judiciously" into large, monolithic blocks of code that have everything "at a glance" in one place?)

orthecreedence8• 11 years ago

After programming in CL for several years, I can honestly say that when encountering new syntax that people have introduced in an app, it's no harder to follow along with it by reading the macro definition than encountering an unknown function. And when in doubt, you can just macroexpand the syntax and see what it's doing under the hood.

PuercoPop4• 11 years ago

I'd say it's probably worth that cost, if used judiciously.

Can be said of anything so it really doesn't say anything at all.

This is could be a good criticism in theory, in practice

you can observe how it works out. Fukamachi uses @ everywhere, attila-lendvai uses his def-star library ej (def (class foo) ...) instead of (defclass foo ...), some people use iterate, etc and all is good.

People can write shit code sans macros no problem. I'd rather have them than don't. After all macros embody the whole point of programming[0]

[0] http://axisofeval.blogspot.com/2011/05/why-of-macros.html

sjolsen3• 11 years ago

When reading other people's code you now effectively have to learn what "language" they use too

And? This is true of code in languages without macros, too. You have too learn all the vocabulary, all the types they use, all the functions, all the structure of the program— macros are just a tool for taking these domain-specific things and packing them into a denser syntactic abstraction. A well-written macro improves readability in both the short and long terms.

I mean, say I write a couple of macros to deal with a database and I write code like

    (with-sql-connection ("localhost" ...)
      (select *
         from  "customers"
         where (= "lastname" "Smith")))

Do you really think that's less readable than the nonsense you'd have to write a typical "framework," or worse, building the command string by hand?

WalterGR6• 11 years ago

    When reading other people's code you now effectively
    have to learn what "language" they use too.

That's well said, and in fact it's not uncommon to talk about Lisp's capacity for crafting the language to solve the problem at hand.

Paul Graham's essay "Programming Bottom-Up" explains it really really well:

"Experienced Lisp programmers divide up their programs differently. As well as top-down design, they follow a principle which could be called bottom-up design-- changing the language to suit the problem.

In Lisp, you don't just write your program down toward the language, you also build the language up toward your program. As you're writing a program you may think 'I wish Lisp had such-and-such an operator.' So you go and write it. Afterward you realize that using the new operator would simplify the design of another part of the program, and so on. Language and program evolve together. Like the border between two warring states, the boundary between language and program is drawn and redrawn, until eventually it comes to rest along the mountains and rivers, the natural frontiers of your problem.

In the end your program will look as if the language had been designed for it. And when language and program fit one another well, you end up with code which is clear, small, and efficient."

http://www.paulgraham.com/progbot.html

    On the other hand, this really hurts readability.

As others have pointed out, no more than unfamiliar classes and so forth.

In any language, you can think of programming as building mini languages to solve problems. You have nouns and verbs - types/classes/instances and methods/functions/operators. Lisp gives you those parts of speech and also lets you manipulate the fundamental grammar as well!

joonoro• 11 years ago

When reading other people's code you now effectively have to learn what "language" they use too.

Paul Graham addressed this in On Lisp (page 59) [0]:

So yes, reading a bottom-up program requires one to understand all the new operators defined by the author. But this will nearly always be less work than having to understand all the code that would have been required without them. If people complain that using utilities makes your code hard to read, they probably don’t realize what the code would look like if you hadn’t used them.
Bottom-up programming makes what would otherwise be a large program look like a small, simple one. This can give the impression that the program doesn’t do much, and should therefore be easy to read. When inexperienced readers look closer and find that this isn’t so, they react with dismay.
We find the same phenomenon in other fields: a well-designed machine may have fewer parts, and yet look more complicated, because it is packed into a smaller space. Bottom-up programs are conceptually denser. It may take an effort to read them, but not as much as it would take if they hadn’t been written that way.

[0] http://www.paulgraham.com/onlisp.html

PuercoPop4• 11 years ago

Just to expand on your comment. One can write the equivalent code for a for in python[0] but one can't make python rewrite itself.

[0] https://gist.github.com/PuercoPop/9d192f94f88074d06625

Foxboron• 11 years ago

cough I'd love to say it's possible, but with a few modifications.

https://github.com/hylang/hy

http://docs.hylang.org/en/latest/language/api.html?highlight...

ra88it• 11 years ago

The act of describing what you want the machine to do is interleaved with the machine actually doing what you have described, observing the results, and then changing the description of what you want the machine to do based on those observations. There is no bright line where a program is finished and becomes an artifact unto itself.

Author doesn't mention macros explicitly, but I think the above quote is referring to them implicitly.

don_draper• 11 years ago

It would help if I saw a real world problem and how you can solve it Lisp and not in say Python.

kazinator15• 11 years ago

Any real world problem solved in Lisp, relative to being solved in Python, eliminates the superfluous problem of Python being involved: a syntactically Fortran-like cumbersome scripting language with somewhat Lispy semantics.

Note that you can program the CL system by expressing yourself in Python:

https://common-lisp.net/project/clpython/

Common Lisp is not only a language, but also a platform (analogous to Mono or JVMs). It has a model of computation: programs expand fully down to a set of special forms, which then compile. You can build languages on top of this.

typedweb3• 11 years ago

How about syntactic extensions:

    (defmacro unless (test &body body)
      `(if (not ,test) ,@body))

parfe• 11 years ago

The best you'll get are examples of something solvable in Python being "beautiful" in Lisp. Then some real world Lisp examples will be references to a 20 year old storefront generator and the initial release of reddit.

Lisp(s) are certainly better than Python in every way, except when it comes to successful projects completed.

gknoy2• 11 years ago

I don't think there are very many "real world" problems that you CAN solve in Lisp, and CANNOT solve in Python. Anything [0] you can write in Python, you can likely solve in Lisp, probably in similar line count (without macros).

One thing I __really__ miss from working in lisp is the idea that I can reload things in the repl. In Python, once I've imported something, I can't really redefine it without pasting in the definition, which makes iterating on a class definition much harder. In Lisp, I can hit a key in my editor and the running REPL gets the new definition, and I can start working with it (or rewriting tests, etc).

The power of the REPL in lisp is __amazing__. I love me some IPython (it's awesome), but there just isn't the same tight integration between that and a running system. The "default" Python likely has all the tools you need to do that, but it just isn't presented as the Way you Do It.

Django's auto-reload when code changes is an example of this. The trouble is, I can't get to a REPL easily within that, without invoking ipdb. I don't know how to integrate my editor with the Python process in a similar way, etc.

All that said, I still love coding in Python. Hy makes me excited, but then I just started writing python-with-parens and wasn't sure what I had gained. :)

0: I'm sure there is someone who can give a counterexample, but I cannot thnk of any.

TeMPOraL2• 11 years ago

All Turing-complete languages are equivalent and can solve the same set of problems. The issue is not whether you can or can't do it, but how much work it will take and how painful the process will be.

Also when you learn some Lisp, you start to realize how often people are greenspuning[0] things in their projects.

[0] http://en.wikipedia.org/wiki/Greenspun%27s_tenth_rule

swuecho3• 11 years ago

there is two sides.

probably, one evil thing is you have to deal with every other's abstraction and it will make you frustrated when there is a large codebase and a deadline.

kazinator15• 11 years ago

Let's see, what would I rather deal with.

* Another programmer's 15 function API, and a document on how to use it properly?

* Or another programmer's 15 function API, along with three macros which use the API properly, and capture all the scenarios I need based on a couple of examples.

Xophmeister• 11 years ago

Could someone please explain the difference between Lisp macros and, say, languages that have first-class functions? I get that a Lisp macro will be expanded into the respective code, while a function's execution is different. However, at the practical (i.e., developer's) level, are there any additional benefits?

Can, say, a Lisp macro be 'partially formed', in the sense that it can expand into some boilerplate that represents an incomplete syntax tree? (Whereas a higher-order-function is necessarily complete.) I can see that being useful, but not inasmuch a it's made out.

typedweb3• 11 years ago

A lisp macro is a code transformer run by the compiler using the full features of the language to generate code.

Macros allow for the arbitrary evaulation of it's arguments (rather than the standard left to right order before a function call), allowing you to do syntactic extensions without the added boilerplate that functional languages can require.

Essentially, each macro allows you to define a mini-language that is parsed by the compiler that returns code that is then compiled. It takes a while to groc, but once you do you can never really go back.

coryrc• 11 years ago

Macros are run before your program is running with actual data (during "compile", if you will).

The advantage of CL macros is you can use functions written for your program to use at run-time during compilation-time/macro evaluation too.

dfan• 11 years ago

One qualitative difference in expressive power comes from the fact that functions first evaluate all their arguments and macros don't.

ThrustVectoring• 11 years ago

This is only ~90% likely to be correct, since it's second-hand information and I don't use LISP actively.

A LISP macro is a syntax transformation. It lets you write code in the way you want to, instead of whatever level of abstraction you used to have.

I'm not sure about 'partially formed' code output, but 'partially formed' input is definitely possible. The way to invoke a LISP macro need not be valid LISP. In short, LISP macros excel at creating domain-specific languages.

beeworker• 11 years ago

One difference is flow control. When you call a function, all the arguments are evaluated before being passed into the function. If you want to delay evaluation, you have to wrap the argument values in a function. When you call a macro, the text forms get passed with no evaluation.

Say Clojure forgot to ship with the boolean "or". "or" should evaluate its arguments one at a time (to allow for short circuiting) and return the first non-false value. You could do this with functions, but you have the source-level overhead of manually wrapping everything, and performance overhead of defining and passing an anonymous function for each argument. With a macro, at compile time you just translate the "or" macro into a simpler form that uses "if". This is how Clojure actually implements "or":

    (defmacro or
      "Evaluates exprs one at a time, from left to right. If a form
      returns a logical true value, or returns that value and doesn't
      evaluate any of the other expressions, otherwise it returns the
      value of the last expression. (or) returns nil."
      {:added "1.0"}
      ([] nil)
      ([x] x)
      ([x & next]
          `(let [or# ~x]
             (if or# or# (or ~@next)))))

chaoky3• 11 years ago

A higher order function doesn't serve the same purpose as a macro. A higher order function is meant to be applied, called, composed etc. A lisp macro is a different type of abstraction. For example, many people think that macros are just hiding lambda's of higher order functions. This is wrong. A macro abstracts over implementation details of a construct to make it read naturally. For example, you can write a function that opens and then closes a file like so...

  with-open-file(filename, lambda file: do stuff with file)

but with a lisp macro, you only have to write

  with-open-file (filename):
     do stuff with file

The point of an abstraction is so you don't have to think about the implementation. Written like the latter, with-open-file is simply more natural to write this way. You only have to think, "oh, its a construct that opens a file then closes it after the body is done", rather than "oh, its a higher order function that i have to pass another function into that takes the file as an argument..." etc.

When you write with-open-file as a macro, it could be implemented as a higher order function, or it could be implemented as a low level set of GOTO statements. It doesn't matter. The macro abstracts away the low level detail, just providing the most natural way for you to use the construct. It might not seem like the macro is doing much in that particular example, but a construct that defines a class (like defclass) is something you can write as a macro, which can expand into functions that do the actual defining. You could write a class defining construct as a higher order function, but then you'd have to constantly worry about how your construct was implemented as a function. instead of just writing something natural like

  defclass tiger (animal):
    age init-value: 0
    name type: string

which you could write if you implemented defclass as a macro. otherwise, you'd have to do something crazy stupid like

  defclass(name='tiger', inherits-from = find-class('animal')
           slots=['age, name'] ...)

how could a higher order function possibly implement that without making people using it tear their hair out? They have to bend to the implementation, not make the abstraction bend to what's natural.

kazinator15• 11 years ago

TXR Lisp is completely strictly evaluated, like many other Lisp dialects. Function argument expressions are reduced to their values, in left to right order. Then the application of the resulting values to the function takes place.

Yet, with macros I have this:

  ./txr -p '(mlet ((x (lcons 1 x))) x)'
  (1 1 1 1 1 1 1 1 1 ... nonterminating sequence of 1's ...

This created a circular list!

The special mlet ("magic let" or "mutual let") construct has allowed the expression which initializes s, (lcons 1 x), to refer to x.

This works because both mlet and lcons are macros. The lcons macro (rather, the code generated by the macro!) returns a lazy cons cell, without immediately evaluating its arguments 1 and x. In the case of x, this is a damn good thing because x is not yet initialized! When the lazy cons is accessed (when the list object is printed), the evaluation of x takes place. By that time, x holds the lazy cons cell, and since the variable x is in scope of the argument x in (lcons 1 x), the lazy cons is able to force, setting its CDR field back to itself, creating not a lazy list, but a circular list.

With lcons, I we can make a fibonacci function quite similarly to how you might do it in Haskell:

   (defun fib2 (a b)
     (lcons a (fib2 b (+ a b))))

We call this as (fib 1 1) and it gives us a lazy list.

Here, we have a marriage between a lazy data structure and a macro. Without lazy conses, the lcons macro would have no target language to expand into. Without the lcons macro, lazy conses can't be used in the above convenient way; we would have to write fib2 in terms of the macro expansion:

  $ ./txr -p "(sys:expand '(defun fib2 (a b)
                             (lcons a (fib2 b (+ a b)))))"

(defun fib2 (a b) (make-lazy-cons (lambda (#:lcons-0001) (rplaca #:lcons-0001 a) (rplacd #:lcons-0001 (fib2 b (+ a b))))))

The circular list mlet, when fully expanded looks like this, by the way:

  $ ./txr -p "(sys:expand '(mlet ((x (lcons 1 x))) x))"
  (let (#:g0001) (sys:setq #:g0001 (cons 'sys:promise
  (cons (lambda () (make-lazy-cons (lambda (#:lcons-0046)
  (rplaca #:lcons-0046 1)
  (rplacd #:lcons-0046 (force #:g0001))))) '(delay (lcons 1 x))))) (force #:g0001))

It's a gritty oat-meal of delays, lambdas, forces, and cons manipulation. One thing that is conspicuously absent amid the toenail clippings: what happened to the x variable? Haha!

snikeris2• 11 years ago

My favorite example for this is the lame idiom you see in Java code:

    if (log.isDebugEnabled()) {
        log.debug("expensive" + debug + message);
    }

This is "better" than just log.debug(...) because with the latter, your expensive log message argument needs to be evaluated even if debug is disabled.

However, in a language w/ macros, you just say:

    (debug (str "expensive" debug message))

and these considerations are already taken care of for you:

https://github.com/clojure/tools.logging/blob/master/src/mai...

crististm• 11 years ago

The sheer number of answers you've got should tell you that indeed there is something about those macros. But you need to 'get' them yourself to appreciate them.

Each answer exposes one or more facets of macros. Indeed, there is not just 'one' thing that make them worthwhile.

aidenn0• 11 years ago

The majority of macros I write could be represented with HOF and lexically-closed lambdas. That adds significant extra syntax when you use them though.

Consider a classic pattern of a with macro:

    (with-mutex-held-macro (some-mutex) (do-stuff))

    (call-with-mutex-held-hof some-mutex (lambda () (do-stuff))

A minor advantage is that a macro will be expanded in-line; a Sufficiently Smart Compiler could transform the HOF version into something equivalent, so it's not strictly an advantage (except to compiler implementers I suppose).

I think that e.g. generalized references (i.e. the common lisp setf macro) are not possible with HOF, but HOF is more common in pure languages where assignment is eschewed anyway.

Mark Jason Dominus (author of Higher Order Perl) had some good comments on lisp macros as well:

http://lists.warhead.org.uk/pipermail/iwe/2005-July/000130.h...

ScottBurson5• 11 years ago

It's true, as others have explained, that for many of the most common uses of macros, you can get the same effect with a higher-order function.

But since macros operate at the syntax level, they can do things functions can't do. For example, they can generate and manipulate declarations. Say you're working with abstract syntax trees (assembly trees in a CAD app might be another example). These trees are built from nodes, where each node is of some class corresponding to a syntactic construct: if-statement, addition-expression, etc. etc. There's some functionality you want to have on every node class; a common example is a "children" method that gathers up all the node's child slots into a set and returns it. It is very convenient to have a 'define-node-class' macro that automatically generates the 'children' method, so that when you add a child slot, the method is updated automatically; there's no need for manual effort to keep them in sync.

In this case, the macro is expanding to multiple top-level declarations: the class declaration along with the method declaration (probably, in practice, several methods). Higher-order functions don't begin to let you do stuff like this.

vorg• 11 years ago

Macros can provide a lot of syntactic convenience over those first-class functions, especially with heavily nested structures. For example, I can replace this monadic parser definition...

    function SpecDeclP()Parser{
      return Bind(SeqRight(MyKeywordP, IdentifierP), function(nm interface{})Parser{
        return Bind(BetweenParensP(IdentifierP), function(parm interface{})Parser{
          return Bind(SetSwitchUserStateP, function(_ interface{})Parser{
            return Bind(BlockP, function(bod interface{})Parser{
              return Result("func " + nm.(string) + "(" + parm.(string) + ")" + bod.(string))
            })
          })
        })
      })
    }

...with this more readable version if I have a recursively defined "macro" called doparse...

    doparse{
      nm   <- SeqRight(MyKeywordP, IdentifierP)
      parm <- BetweenParensP(IdentifierP)
      SetSwitchUserStateP
      bod  <- BlockP
      Result("func " + nm.(string) + "(" + parm.(string) + ")" + bod.(string))
    }

This requirement arose in Haskell for its Parsec before it became a part of the syntax, then again later with the Arrow library which later also added it to the syntax. Whenever new abstractions are discovered/created, a macroing facility, whether for lisp-like syntax or some other, helps makes all the nested functions more readable.

calibraxis• 11 years ago

You can support new paradigms, just as a library. For example, core.async was written as a macro. So Clojure basically got Go channels in a library, not a language upgrade.

Macros can fundamentally change code. Codewalkers. Back when OOP was a big thing (probably still is), Lisp programmers would probably amuse themselves by making object-oriented extensions to the language via macros, and sending them to each other.

This means that you are empowered, not just a language implementer. (First-class functions are important, and typically a better idea than "Guess I'll write my own macro!" but they only go so far.)

jlarocco2• 11 years ago

Your question doesn't make sense. High order functions and macros are orthogonal concepts.

Lisp supports first-class, high order functions. In fact, it was probably the first language to do so.

Macros are different. And they're not just "expanded." Think of macros as full blown Lisp functions that run at compile time and generate new Lisp code, that is itself compiled at compile time. Since Lisp code is represented as Lisp's list data structure, it's super easy. The macro system does allow expansion/replacement (like C's preprocessor), but that's just scratching the surface.

TuringTest• 11 years ago

The first order function is evaluated at runtime, every time the program is executed. Macros are expanded at compile time, so they will be calculated just once. Thus the syntactic sugar added by the macro doesn't incur a time penalty.

This can provide an important speed improvement for complex macros or code used often, in tight loops or frequently called functions (it's like using inline methods in .h files in C++).

dschiptsov3• 11 years ago

The difference is subtle. With high order function you coule define (abstract out) a new idiom, while with proper macros you could define a new special form.

In case of using high-order procedure all its arguments will be evaluated in order before application of the procedure while in case if a macro you could explicitly define all the transformations (evaluation rules) for each argument. This is why it is called a special form - it has its own evalustion rules.

Shortcircuiting if or and are canonical examples.

So, with macros you are extending Lisp with new special forms.

dorfsmay2• 11 years ago

Aren't macros breaking the homocionicity of the lisps?

And making maintenance more difficult (aren't you re-inventing a new language with macros?)?

Where does the "First rule of the macro club" coming from? When should you break it?

dragonwriter2• 11 years ago

Aren't macros breaking the homocionicity of the lisps?

No. How would that be the case?

And making maintenance more difficult

If used poorly, sure; the same is true of most features of most languages.

(aren't you re-inventing a new language with macros?)

You are extending the existing language with them, much as you are when using a language with user-definable functions rather than only builtins.

sjolsen3• 11 years ago

Aren't macros breaking the homocionicity of the lisps?

No. How would macros break homoiconicity? They expand into atoms and lists (and other datatypes), the same stuff of which macro-free programs are made.

And making maintenance more difficult?

No. Unless you intentionally write unmaintainable macros. It's the same as if you write unmaintainable functions, classes, etc. They're just abstracting a different thing— syntax.

aren't you re-inventing a new language with macros?

No, you're expanding the language; you're just expanding the grammar instead of the vocabulary.

Where does the "First rule of the macro club" coming from? When should you break it?

What is the "first rule of the macro club?"

xamuel2• 11 years ago

maintenance

If used right, macros make things much more maintainable. If you have a hundred nearly-identical codeblocks, where only (say) a string constant is varying, and suddenly you need to change what those blocks do, bam, you've got a hundred blocks to change. If those blocks had been refactored with a macro, all you have to do is change the macro once. It also prevents the new guy from coming and doing a bandaid fix to line #72 making it slightly different from the 99 lines around it and making your life a living hell.

GFK_of_xmaspast8• 11 years ago

It's kind of cool how in lisp you're supposed to do macros to change major aspects of how the language behaves, but in C if you try having a little bit of fun with #define and the pre-processor everybody starts getting just extremely rude at you.

swuecho3• 11 years ago

That is part of the reason. everybody think lisp cool, but very few programmer use lisp.

the_af8• 11 years ago

To be fair, C #define macros are extremely dangerous and error prone in ways that Lisp macros aren't.

I haven't bought into this "Lisp macros are the best thing since sliced bread" idea, but without a doubt they are infinitely more awesome than C macros.

Dewie32• 11 years ago

Could the reason perhaps be that Lisp macro facilities tend to be much more sane than C's pre-processor? Just a wiiild guess.

stephengillie2• 11 years ago

I'm sorry for asking a dumb question; I'm a sysadmin not a software developer:

Is this similar to needing a dozen nearly-identical lines of code, writing one, then using Excel to manipulate the other 11 lines into what you want, then copying it back (and cleaning tabs in the process)?

Essentially programmatically writing the program - is that what Lisp's macros are?

bliti• 11 years ago

More or less. It allows you to write generic code you can reuse. In this case, you are extending the language. Feel free to post these "dumb" questions, or send them my way.

willismichael• 11 years ago

Not a dumb question at all, and yes, that's more-or-less what Lisp macros are, albeit more elegant because you're using the same language to write your code as you use to write the macros that manipulate the code.

WalterGR6• 11 years ago

    Essentially programmatically writing the program -
    is that what Lisp's macros are?

Macros are often described that way. "Lisp macros let you write code that writes code!" While that's a true statement, it's kind of useless for coming to an initial understanding of what macros let you really do.

Think of it this way. Programming languages have built-in control flow operators like if-then, do, while, and for. Macros let you write your own operators - that work just like the ones built into the language itself.

Here's an example. Let's say you want to print a list of names.

In a language without an operator specifically for going over lists - and without macros - you may have to write:

    ListIterator iterator = namesList.GetListIterator();
    while(iterator.HasMoreItems()) {
        string name = iterator.GetNextItem();
        print(name);
    }

If you have a language that supports macros, you can create a macro called forEachItemInList. And then it'll work just like it came with the language all along:

    forEachItemInList(string currentName in namesList)
    {
        print(currentName);
    }

If you count how many words the non-macro solution took, it's 13. The macro solution takes 7. Having to solve a problem with more code means more time to understand, teach, write, test, debug, and document - so savings like this can really add up.

And there's also the subjective part. Having to write code like in the first example just isn't satisfying. You have to repeat yourself constantly. You may think, "Look, I loop over lists all the time. And it's always the same pattern: I call GetListIterator(), I do a 'while' loop while it HasMoreItems(), then I call GetNextItem(), etc. Why can't I just inform the compiler of the general pattern - and then tell it how to fill in the blanks when I need to actually loop over a list?"

And that's what macros let you do.

btilly• 11 years ago

Yes, except more reliable.

The fundamental unit of Lisp is expressions of the form (FOO ...). There are three basic cases.

1. FOO is one of a finite list of special forms. Those are the irreducible base of that language.

2. FOO is a function. In which case the ... is evaluated as Lisp (possibly including doing things like expand out other functions) before being passed to that function.

3. FOO is a macro. In which case the ... bit is turned into a list of things, but NOT evaluated, passed to the macro to be rewritten, and then the result is turned into an expression that is then evaluated by the same rule.

This is not completely true. There is a preprocessing step called reader macros that can do things like turn `(this expression) into (QUOTE this expression) to save typing. But those are very sparingly used.

hzhou321• 11 years ago

I agree. Macros are what made lisp, not parentheses.

However, if we broaden the concept a little, we can do this type of macros for any language with a meta-layer such as MyDef. e.g. if you observe certain foreach pattern in Java, you can make a macro for that pattern and have the meta-layer look out and translate that for you (like having an automatic translator between you and javac. All you need is a scope type macro. Conventional macro packages like M4 does not provide such, but MyDef does. Example:

    &call each_member, AList
        # java code that work on $(member)

where the macro may be defined as

    subcode: each_member(list)
        Enumeration e = $(list).elements();
        while (e.hasMoreElements()){
            String name = (String) e.nextElement();
            $(set:member=name)
            BLOCK
        }

Where it is understood that the definition can be any literal block pattern.

WalterGR6• 11 years ago

    Macros are what made lisp, not parentheses.

The parentheses are huge, though. It enables "Code is data" and "Data is code" in a powerful way.

About Lisp, people always say "Macros are great" and "Code is data" and "Data is code", but it's hard to see what they mean without good examples. I mean, you can write code that writes code in any language that has a `print` statement. And obviously code is data, so what is that aside from some pseudo-philosophic BS?

There's a lot of discussion in this thread about macros - so here's an example of Code and Data Being One, for those that are unconvinced. I hope it'll shed some light.

I have a slang dictionary website. It's backed by a database now, but it used to be statically-generated HTML. I represented the data as XML. It looked something like this:

    <Term term="slick">
      <PartOfSpeech pos="adj">
        <Definition>
          impressive.
        </Definition>
        <Definition>
          smart.
        </Definition>
      </PartOfSpeech>
    </Term>

So then I needed an XML parser to parse the data. Maybe it parsed the XML into a object model that the code would navigate and output the appropriate HTML. Or maybe the code got callbacks according to node type and would output the appropriate HTML then.

XML is a pretty verbose format, so - this being a Lisp example - we could probably save some typing if we represented the data as an S-expression. The above XML would become something like this:

    (Term "slick"
      (PartOfSpeech "adj"
        (Definition
          "impressive."
        )
        (Definition
          "smart."
        )
      )
    )

But that's an idiosyncratic style.

So, let's lowercase the node types, use hyphens rather than camelCase, and remove the unnecessary line breaks. Now the data looks like this:

    (term "slick"
      (part-of-speech "adj"
        (definition "impressive.")
        (definition "smart.")))

Great. We've got our data.

Now we need to write the Lisp code to convert that S-expression data into the appropriate HTML output.

We'll need some Lisp functions to handle the nodes and their attributes (such as the definition text and part of speech) and write the HTML for them. We could use an S-expression parser library to load the data, and then walk through it and call those functions. But that's not necessarily the best way. We can simplify it by creating exactly 3 functions that take some arguments and output HTML: term, part-of-speech, and definition.

Since Lisp code is - like the data - also represented as S-expressions, once we've written those 3 functions, the data is literally executable Lisp code.

larme• 11 years ago

Macros are great.

I'm writing a hy-like lisp language that compiles to lua. A lot of features (Simple OO, list/dict comprehensive etc.) are written in macros.

https://github.com/larme/hua/tree/master/hua/core

On the other hand, they may be difficult to debug.

agumonkey2• 11 years ago

Note that this is a double edged sword. Everyone is having is own little DSL world. A thing that Lispers often enjoy, but it's an art, a beautiful one, and a beautiful freedom too, and most people want a python with easy to access libraries more than that.

Lisp aims itself into niche problems where there's nothing good enough done yet. For the rest people won't care about the ability to fit the language to their need, they want 'productivity' and if a pattern become useful, backpressure will force language designer / BDFL to change it.

hueving2• 11 years ago

So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years

I get the sentiment, but this is seriously advocating for unmaintainable code that likely breaks in strange corner cases and requires 10 times as long for a new person to the code base to understand.

First contributor, "I know my code should be able to handle out of memory errors in this corner case, but I know I won't use it that way so it shouldn't be an issue."

Second contributor, "Hey this macro from First contributor looks nice, I'm going to use it."

A pile of shit emerges.

badpenny• 11 years ago

A pile of shit emerging whenever programmers are lazy and try to cut corners or when they try to do things they don't fully understand is not unique to macro writing.

current_call• 11 years ago

The book you linked has examples of macros being used for writing a music database, parsing binary files, a unit testing framework, and several other things I haven't reached yet. Those are much better examples of macros than basic control structures. The 'dolist' form is a toy example.

renox2• 11 years ago

Similar, similar, only if you don't look too closely: what about break and continue? They are usually supported by native foreach construct but are difficult to build with macros..

WalterGR6• 11 years ago

    only if you don't look too closely

Look as closely as you'd like. :)

    what about break and continue?  They are usually
    supported by native foreach construct but are
    difficult to build with macros..

For break, DOLIST uses return. I haven't worked with Lisp in some time, so I don't remember if it has a dedicated continue. But a continue is just a break in a loop that doesn't loop. Or a GOTO by another name. Or a jump to a particular case in a switch.

But if your looping macro uses a native looping construct or another macro that supports break and continue, then your looping macro will inherit that support, provided that you're careful when writing the macro to do nothing that will break it.

If your looping macro doesn't use a construct that supports break and continue, then you're still in luck. Continue I described above. Like continue, break is also GOTO by another name. You've got TAGBODY, GO, CALL/CC, etc.

Why do you think that supporting break and continue in macros is difficult?

dang• 11 years ago

Among other things, it makes writing interpreters and compilers really easy, and so inventing new languages and writing interpreters and compilers for them becomes [...] a part of day-to-day Lisp programming.

This matches my experience. The barrier between application and language/compiler development disappears, and instead you get a rich new feedback loop between the two. Overall complexity decreases (<-- a big deal), and many hard things become easy.

The effects of this compound over time, so what starts as a mere notational difference turns into a deeply different programming style, one I find so enjoyable that doing without it feels like going back into a straitjacket after escaping.

pjmlp• 11 years ago

One issue with many coders not getting Lisp, is that environments open source environments feel short of classical Lisp enviroments like LispWorks and Allegro CL 9.0.

Having used the old Smalltalk and Oberon environments, sadly not Lisp ones, I really thing many still don't get it.

myth_buster• 11 years ago

Here's [0] PG's take on it.

[0] http://lib.store.yahoo.net/lib/paulgraham/jmc.ps

serve_yay• 11 years ago

There are certainly good reasons why lisp is the way it is. I dislike reading that style of code, though. I don't like how you have to read it in a strange sort of top-to-bottom-but-also-inside-out way (really this happens in all languages but it's especially bad in lisp because there are no infix operators and so forth), and of course the common gripe about all the parens.

And to be honest, I am not really interested in how hard it is to write a compiler for the language I use.

weavie3• 11 years ago

Clojure has a threading macro built in (http://clojuredocs.org/clojure.core/-%3E) which lets you write code like :

    user=> (-> "a b c d" 
               .toUpperCase 
               (.replace "A" "X") 
               (.split " ") 
               first)

It is trivial to write the macro in other Lisps.

sgrove• 11 years ago

This is one of the nicest macros - chaining with sane return semantics and no overhead/boxing required. Breaks my heart trying to replicate it.

_random_• 11 years ago

Death by 1000 nested round brackets.

platz3• 11 years ago

S-expression syntax is a well-designed serialization format

I must press, well-designed according to what measure? OP presents the well-designed serialization format as a strict positive instead of the truth: a tradeoff compared to other unnamed qualities.

munificent• 11 years ago

The reason that code represented as XML or JSON looks horrible is not because representing code as data is a bad idea, but because XML and JSON are badly designed serialization formats.

By that same token, a Volkswagen Beetle is a badly-designed boat.

XML was never designed as a data serialization format. It's a markup language. It was designed to sprinkle structure and metadata into large human-readable plaintext documents.

Likewise, JSON is a subset of a general-purpose programming language's literal notation that happened to be very fast to parse in a browser by virtue of the browser implementing that language.

Personally, I don't think s-exprs are a particularly great serialization format either. The problem is that there's no one-sized-fits-all for serialization. What we value is brevity, but basic information theory tells we can only make expressing some things more terse by making others more verbose.

When you say some format is badly-designed, all you're really saying is that it isn't optimized for the kinds of data you happen to want to serialize.

lisper7• 11 years ago

XML was never designed as a data serialization format. It's a markup language.

Those two things are not mutually exclusive.

Likewise, JSON is a subset of a general-purpose programming language's literal notation that happened to be very fast to parse in a browser by virtue of the browser implementing that language.

That's true. That is not in conflict with anything I said.

The problem is that there's no one-sized-fits-all for serialization.

No, that's not true. S-exprs really are a global optimum in the space of serialization designs. All the alternatives are logically equivalent to S-exprs but with extra punctuation that makes them arguably harder to read, but inarguably harder to write. That is why S-exprs are the ONLY syntax ever designed (some would say "discovered") by humans that has been successfully used to represent both code and data.

CJefferson2• 11 years ago

Is there any loss less binding of XML to s expressions? I've never seen one.

Usually example where I see people rewrite XML to s expressions (like in this thread) are very lossy -- its easy to be pretty by throwing away most of the information!

ScottBurson5• 11 years ago

> XML was never designed as a data serialization format. It's a markup language.
Those two things are not mutually exclusive.

I beg to differ. I just replied to someone else about this: https://news.ycombinator.com/item?id=9509110

I agree with your last paragraph, though. There is a timelessness about S-expressions.

As a side point, I would add that the distinction between strings and symbols is important, and neither XML nor JSON has it.

comex• 11 years ago

One downside to S-exprs compared to, say, JSON: they do not have direct support for unordered mappings (hash tables, dictionaries, whatever you want to call them). You can represent them as trees, but basically every language these days (including, of course, Lisps) has a mapping type as a core concept; requiring the user to figure out what parts of the input data should be converted to that type is annoying, and makes the format less self-documenting (i.e. it may not be immediately apparent whether there can be duplicate keys or not).

http://eli.thegreenplace.net/2012/03/04/some-thoughts-on-jso...

AnimalMuppet• 11 years ago

What we value is brevity, but basic information theory tells we can only make expressing some things more terse by making others more verbose.

But the reverse is not necessarily true. It's possible for a badly-designed serialization format to be longer in all cases than some other format.

And, in particular, I suspect that for the same data, XML is longer in all cases than S expressions.

albatross_down• 11 years ago

Cached version here: http://webcache.googleusercontent.com/search?q=cache:KvLXVU9...

pshc• 11 years ago

Nah, Lisp is nowhere near the most minimal way to serialize code. It's plaintext, so it's denormalized. Something akin to bincode is the most minimal. But that requires a non-plaintext editor...

smizell2• 11 years ago

I actually created a hybrid of JSON and Lisp called Geneva [1]. It was a fun weekend experiment. I totally relate to how the author feels

[1] https://github.com/smizell/geneva

PuercoPop4• 11 years ago

Early versions of wat-js used ["the", "same", "syntax"]. You might find it interesting. https://github.com/manuel/wat-js

smizell2• 11 years ago

Ah, very interesting! Thanks for the link.

dschiptsov3• 11 years ago

There are whole books which answers that question. The one I am aware of is pg's "On Lisp" which explains everything clearly without any "Haskellish or monadic mysticism".

There is also the famous essay "Beating an average" and this very site as a walk of their talk.

There are also SICP and PAIP and Norvig's Lisp Style Guide.

Read the best thinkers.

dschiptsov3• 11 years ago

My 2₹ - http://karma-engineering.com/lab/wiki/Bootstraping

reddytowns• 11 years ago

For me, I think the most important reason for working with lisp and its variants is that it allows you to (and you should) code from the middle.

When I first started programming, and for a long while, I'd think top down. What do I got? Then, how can I iterate through that to do the operations I want and then build up what I need?

With lisp, the ability to reuse code in closures is so easy, you can afford to think a different way. What are the most basic, clean and simple, operations I need to do? (not necessarily in order I need to do them) This includes conversion, summing results, iterating through a structure, etc.

After writing functions for those, you can quite easily plug them all together without much thought.

You end up with code that is non-redundant, clear, reusable, easy to debug, and flexible.

igravious3• 11 years ago

(I have a subtle optimization for S-expression syntax)(I am surprised nobody ever thought of it)(When S-expressions are in a sequence use an extra (special) delimiter plus the regular token separator to separate expressions)(Maybe use dot? (period I think some call it))

Like so. I think it could catch on. And you get rid of so many round bracket block delimiters (at least for S-expressions on the same level. for nesting you obviously need them) that it makes reading a lot easier. Also make the language modal and have the default be the indicative mood. Maybe replace "." with "?" for interrogative mood? Maybe elide ".)" to ")" as another optimization?

lisper7• 11 years ago

You'll love my binding-block macro then. You can find it here:

https://github.com/rongarret/ergolib

igravious3• 11 years ago

Interesting. Now all I have to do is learn a bit of Common Lisp. I'm surprised my half-jokey question turned out to have a legitimate response :)

roryokane3• 11 years ago

So this code (from https://github.com/axch/test-manager/blob/c4fc3224e873716c58...)

    (define-record-type omap-entry
      (make-omap-entry key item next prev)
      omap-entry?
      (key omap-entry-key set-omap-entry-key!)
      (item omap-entry-item set-omap-entry-item!)
      (next omap-entry-next set-omap-entry-next!)
      (prev omap-entry-prev set-omap-entry-prev!))

would be written like this?

    (define-record-type omap-entry
      make-omap-entry key item next prev.
      omap-entry?,
      key omap-entry-key set-omap-entry-key!.
      item omap-entry-item set-omap-entry-item!.
      next omap-entry-next set-omap-entry-next!.
      prev omap-entry-prev set-omap-entry-prev!)

(To mark `omap-entry?` as being a value, not the S-expression `(omap-entry?)`, I decided to place a comma after it instead of a period. The alternative is requiring no punctuation and making newlines significant.)

Well, I suppose it does look better.

However, the idea is of limited applicability. While looking through the example project (https://github.com/axch/test-manager), I had trouble finding some code where this would actually be useful – most code has too much nesting.

That’s why I prefer another solution for removing excess parens from Lisp syntax: making whitespace significant. It improves the syntax in cases where your periods would help, and it applies in additional cases as well.

“Sweet-expressions” (http://readable.sourceforge.net/) is an implementation of that. Here is the above code ran through the `sweeten` tool to convert it to sweet-expressions:

    define-record-type
      omap-entry
      make-omap-entry key item next prev
      omap-entry?
      key omap-entry-key set-omap-entry-key!
      item omap-entry-item set-omap-entry-item!
      next omap-entry-next set-omap-entry-next!
      prev omap-entry-prev set-omap-entry-prev!

Absolutely no parentheses necessary, while still preserving homoiconicity. You don’t even have to remember the `)` at the end of the last nested line. And I chose this example code to look good with your idea – when the code has more nesting, sweet-expressions look even better.

About the interrogative mood `?` you describe: it might simplify `if` statements. But that would require removing the convention where boolean-returning functions have a name ending in `?`, such as `omap-entry?` in the example. It’s a tradeoff.

igravious3• 11 years ago

Sweet-expressions is a cool idea. Was it Python that started the whitespace-is-significant trend? I'm wondering why someone doesn't create a Scheme or a Lisp with this as valid syntax out of the box. I think you'd still need to support regular S-expressions though, right?

jgrant27• 11 years ago

I haven't been here in a while now but when Ron shows up in the mainstream that's reason enough. Why Lisp ? Definitely the reasons he points out.

I worked at a large e-commerce retailer for a few years, one that you might have heard of or even purchased something from. I had the experience of building a few systems in Lisp and also in a few mainstream imperative languages. It was tough and really lonely even though I had real success.

With Lisp I could do things that whole teams could only dream of in Blub and in much shorter periods of time.

The problems weren't really technical but more psychological and social when it came to Lisp. Big problems. I wasn't very successful in resolving many of them.

I still use Lisp here and there but have been drawn to the ML family in the past few years. Strong static type systems got a hold of me for better or worse.

All languages suck, it's just the degree that differs.

hitlin37• 11 years ago

If the page doesn't download for anyone, here is archive snapshot from today: http://web.archive.org/web/20150508012536/http://blog.rongar...

lessthunk• 11 years ago

Why Lisp? Because the developers that know and love Lisp are normally highly above average and get things done. For sure, there is a much smaller pool to chose from, but once you are far from what is currently hip, you get developers that actually care.

dptd• 11 years ago

Nicely said. Thanks!

jahnu• 11 years ago

What are some of the best known applications written in a lisp?

Why Lisp?

Related Stories