The Problem with Macros

https://ianthehenry.com/posts/janet-game/the-problem-with-macros/

by todsacerdoti • 5 years ago

151 69 5 years ago

BalinKing2• 5 years ago

As a relative Lisp novice, I think this article is excellent! I really enjoyed it, both from a technical and also an entertainment perspective. Time to put the rest of the blog post series on my ever-growing reading list….

One minor typo (I thought it was pretty funny): “it seems like it was actually a performance hack in order to make a dynamically typed language performant on the hardware of the early 1890s.” Or, did Babbage’s machine run Lisp the whole time?

munificent• 5 years ago

I assumed this was a deliberate joke.

foobarian• 5 years ago

It seems like a pretty excellent way to refer to the 1980s.

ianthehenry4• 5 years ago

In his paper "History of Lisp" (PDF warning: http://jmc.stanford.edu/articles/lisp/lisp.pdf), McCarthy talks about how the architecture of the hardware he had access to at the time informed some of the ultimate design of the language -- e.g. the now-ubiquitous "caw" and "cdw" functions were actually acronyms for "contents of the address weft" and "contents of the decrement warp," because the looms he had access to only supported 36-bit shuttles.

(Thanks for the kind words. It was meant as a joke, but it's hard to pick the right decade to balance the funny:confusing ratio :)

BalinKing2• 5 years ago

Yeah, I'm pretty bad at picking up jokes sometimes—it seems this one's on me :-)

LanternLight83• 5 years ago

I shared those with my crafts-inclined partner, who filled me in on what "weft, warp, and shuttle" mean in the context of a loom- I still don't entirely follow, but where-ever this tidbit of computing has it's roots, I love it.

mst4• 5 years ago

People sometimes mistakenly knitpick my jokes, but I always put it down to writing too much purl.

eyelidlessness• 5 years ago

This is quite a wild ride. My only Lisp experience is Clojure, and I have written some macros[1], but never dove anywhere near this deep into everything that’s going on.

It’s interesting, I’ve spent a lot of recent personal interest time deep diving into code transformation in JS/TS. These concerns about name collision and shadowing are front and center there, but of course tree-walking AST transform is the ~only game in town[2] and it’s had me pining for first class language macros.

1: The first that comes to mind https://github.com/reup-distribution/espalier

2: I have some ideas on this, but nowhere near fully formed enough to elaborate just now; suffice to say they derive a lot from what I do know about macros.

srcreigh2• 5 years ago

Bravo, my friend, bravo.

The unit testing macros for my Racket project were created so I could create snapshot tests for an API. The custom test-suite and test-case macros set parameters (think React context properties) for the suite and case names. There is a custom `check-snapshot` function which save snapshot files under ./snapshots/suites/{suite-name}/cases/{case-name}. It evaluates each expression and pretty-prints the results separated by newlines. It can use an env var flag to fail any test if results differ from the file, but I usually just always update snapshots and check in Git to see if/how they have changed.

One day I'll improve `check-snapshot` to copy the unevaluated expressions into the snapshot file too so they're very easy to review. This will require it to become a macro. The evaluated expressions are generally 1-5 pretty-printed lines long and stateful sequences of expressions, so it's different from React snapshots which can be pages long to display one value.

Anyways, I'm unsure if Racket macros suffer from the problem described in the post. If I'm not mistaken, this problem is kinda like a problem with unhygienic macros only? The idea of hygienic macros is that identifiers are never captured from the environment where they're used.

In Racket there's also phase separation. The tests.rkt file in my project actually has two test-suite identifiers. One at phase 0 referring to the custom test-suite macro, and one at phase 1 referring to rackunit's test-suite. Good thing the macro isn't recursive, I guess...

agumonkey3• 5 years ago

Talking about Racket and macrology, I think OP should get in touch with the PLT/Racket team. Matthew Flatt made many talks and papers about macro layers and proper scoping.

srcreigh2• 5 years ago

Parent comment is likely referring to the wonderfully named "Advanced Macrology and the Implementation of Typed Scheme" paper [1]

If you type macrology into Google you get the definition "Long and tedious talk without much substance; superfluity of words. noun." Perhaps accurate, but too vague ;-)

[1] https://www2.ccs.neu.edu/racket/pubs/scheme2007-ctf.pdf

brandmeyer• 5 years ago

Syntactic closures are another hygine system that provides the same construction techniques as common lisp macros, while also allowing the macro author to choose which bindings should be set in the macro's lexical environment instead of the expansion's lexical environment (or vice versa).

http://community.schemewiki.org/?syntactic-closures

ianthehenry4• 5 years ago

Thank you for this. My brain is complete mush after days of reading papers about the history of lisp, but this is at the top of my reading stack once I can get my eyes to focus again.

agumonkey3• 5 years ago

HN not supporting notifications, I am forced (;-) to link you some names/topics that you might enjoy

https://news.ycombinator.com/item?id=28847236

reikonomusha5• 5 years ago

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of Common Lisp code have been running for decades without running into problems with capture. So I purport solving this problem is akin to solving a 0.00000001% issue if we measure the frequency of encountering this error writing thousands of lines of Lisp per day for a sizable chunk of one’s lifetime. The explanations by Pitman et al. are wise. If someone on my team were regularly writing macros that expanded into non-externalizable, non-PRINTable function objects, I’d probably be irritated. You’re hindering my pretty-printer, you’re hosing my debugging tools, and you’re breaking my expectations of interactive development. What? One big negative the article doesn’t cover is that if you’re essentially embedding function pointers in your macros, redefining that function won’t take effect in previously expanded code.

    (defun f () 1)
    (defmacro m () `(f))
    (defmacro n () `(,#'f))
    (defun g () (m))
    (defun h () (n))

    ; later
    (defun f () 2)

H will always return 1 (bad!). G will reflect the global binding (good!). You would have to re-evaluate the source definition of H to get it to reflect the new pointer. This is contrary to Lisp’s DNA of interactive development and is a big no-no. (You have similar issues if you, a library author, are declaring the functions you provide to application programmers as globally INLINE. It’s not for you, the library author, to choose when things should be inlined!) You can force late binding using FUNCALL on a symbol to look up the global binding

    (defmacro n++ () `(funcall 'f))

if you so please, but it’s not very satisfying. F can never be captured by the local environment there.

waterhouse6• 5 years ago

redefining that function won’t take effect in previously expanded code

Yeah, that is an important issue. What do you think of the following solution?

1. The semantics of the language are that, every time a function is called, any macros in its body get reexpanded.

2. The language implementation is expected to notice that, in normal cases, neither the macro nor the functions it calls have been redefined, and therefore the previously-expanded (and -compiled) version can be used; but then to notice when that changes (i.e. when you do redefine something) and generate new code accordingly.

The feasibility of 2 I'm not sure about.

Edit: Regarding the printing of functions, I don't think it would be that difficult to make the pretty-printer, upon encountering a function object, detect whether that function is bound to a global variable, and if so, print the name. (Possibly the name with a "f:" prefix, or in a different color.)

fiddlerwoaroof5• 5 years ago

The data needed for (2) is mostly already available: most lisps maintain a Xref database and Slime already has a binding for `slime-who-macroexpands`[1]. Integrating this performantly into the compiler might be interesting, but I think it's theoretically tractable.

[1] https://common-lisp.net/project/slime/doc/html/Cross_002dref...

reikonomusha5• 5 years ago

By “print”, I mean it in the Lisp sense of “print readably”. In Lisp, PRINT really means “serialize as text that can be deserialized with READ.” In Common Lisp (and Janet), functions print fine in the colloquial sense:

    > (print #'print)
    #<FUNCTION PRINT>

The character sequence ‘#<’ means it’s “unreadable” though.

TeMPOraL4• 5 years ago

1. might fly in some purely-functional Lisp, but most (all?) real-world Lisps can have side effects in macro expansion. And by "side effects", I don't mean altering some global variable or adding a function to the symbol table - Lisp macros have access to the full language runtime, so a macroexpansion may just as well send some HTTP requests or hit a database.

Ironically, the author of the article was correct in their view that he initially described as wrong. Macros are just functions, typically taking code and outputting code. They just didn't understand the evaluation rules surrounding macro invocation.

ianthehenry4• 5 years ago

I purport solving this problem is akin to solving a 0.00000001% issue

It seems like Common Lisp has done quite a lot to solve this problem already, which is why it is a non-issue in Common Lisp.

non-externalizable, non-PRINTable function objects

Yes, this would be a very annoying technique in a language with non-externalizable, non-PRINTable functions.

This is contrary to Lisp’s DNA of interactive development and is a big no-no.

Redefining functions interactively is actually a great motivating case for cross-stage persistence. Instead of interpolating the value of the function, interpolate the lexical binding of the function itself, and then redefinitions will be reflected automatically in any expansion of the macro.

reikonomusha5• 5 years ago

It’s ambiguous to me whether you’re suggesting there are compiled languages with printable and externalizable functions.

Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.

fiddlerwoaroof5• 5 years ago

I strongly agree here: my experience is that a Lisp-2 with namespaced symbols has very few issues with function-name capture. While, at one point, Lisp-2s may have been for performance or other implementation details, I find that Lisp-2s are much nicer to code in because you just don't worry about name collisions (as long as you know a handful of relatively simple rules about macros).

kazinator2• 5 years ago

My experience that a certain systems programming language of Unix origins featuring 1 namespace and an unhygienic token-based macro system also has few issues with function-name capture, even in code bases with seven digit LOCs.

That makes me severely disinterested and skeptical about hygienic macros.

They are too weird for the little benefit they provide. You can't look at a piece of code and know what it expands to.

Hygienic macros break lexical scope, because code produced by a macro being invoked in some file "A.lisp" is seeing lexical variables defined in some different file "B.lisp". Lexical scope must be physically enclosed and contained. If you expand it here, it sees the variables that are here, and not some other ones elsewhere that you don't see here.

Given a nesting like (x (y (z a))), if y binds a, that must be like a brick wall; there is no way that the a reference in z can get around it, to connect with a binding of a produced by x. Either that a refers to z's own binding, or else to y. Yet hygienic macros can perpetrate a lexical-scope-destroying wormhole which can do that: the inner a reference can bypass a definition set up by y, and go to the one in x.

It's a downright security issue, like going around your company's firewall, or escaping a container or chroot jail or whatever.

lukego2• 5 years ago

Similar to variable capture with dynamic scope e.g. in classical Emacs Lisp? That's been the root cause of a bug for me exactly once in years of Emacs hacking, and having finally been bitten my radar detects that risk going forward. So much as I prefer lexical scope the alternative seems pretty tame in practice.

TeMPOraL4• 5 years ago

I miss dynamic scope in non-Lisp languages. Perhaps defaulting to it for general-purpose programming was a bad idea (one on which most Lisps backtracked), but it's arguably a good choice for an extensible application like Emacs, and it's definitely a tool you want to have available.

An easy way to see this is: dynamic binding is to environmental variables what lexical binding is to command line arguments.

jsnell2• 5 years ago

Which is fantastic; very useful backtrace; no notes. And why is this? Why can’t I shadow this function?

Not sure what the author means by "no notes". The transcript they posted does point to both the exact chapters in both the implementation and standard docs:

    ;   See also:
    ;     The SBCL Manual, Node "Package Locks"
    ;     The ANSI Standard, Section 11.1.2.1.2

komon• 5 years ago

"no notes" is an idiom meaning "I have no suggestions for improvement"

The author was being sincere, rather than sarcastic as your initial parse suggests

jsnell2• 5 years ago

I was not suggesting they were being sarcastic. I was suggesting they had not actually noticed that the error message actually answered their question: "Why can’t I shadow this function?".

This is the section of the standard the error message points at: http://www.lispworks.com/documentation/lw71/CLHS/Body/11_aba...

And this is the reasoning for why that section exists, linked to from that page: http://www.lispworks.com/documentation/lw71/CLHS/Issues/iss2...

abbeyj• 5 years ago

The phrase "no notes" is slang from show business. After you're shown a script or a performance you may have "notes" to share indicating what you think is wrong with it. If you have "no notes" then you think it is perfect as-is and needs no changes.

The author here is not saying that the message has no notes. They are saying that they have "no notes" to offer the creators of the message on ways to improve it

See, for example: https://forum.wordreference.com/threads/no-notes.3042292/ https://hinative.com/en-US/questions/15238431 https://www.youtube.com/watch?v=5NVQ8v1P4go

waterhouse6• 5 years ago

I had already, at this point, verified that this is something you can do in Common Lisp. [...] But maybe… maybe you couldn’t do this back then? On Lisp was published in 1993. Maybe… maybe Common Lisp macros were entirely syntactic in 1993? Is there a chance that this is some, like, recent development in the lisp world?

Paul Graham mentioned this (EDIT: something like this, but probably with a different answer) once in http://www.paulgraham.com/ilc03.html :

What happens when a Common Lisp macro returns a list whose car is a function? (Not the name of a function, mind you, but an actual function.) What happens is what you'd expect, in every implementation I've used. But the spec doesn't say anything about this.

(EDIT: I think "what you'd expect" actually means "raising an error" here. Either that, or implementations have changed since 2003, which is also possible.)

I do suspect that this is the nicest way (from a theoretical standpoint: requiring the least additional complexity in the language semantics) to get "macro hygiene". Then you probably want a different syntax from quasiquote, because otherwise I think you'll put commas on the majority of the symbols. But using "list" a lot (and "append" when you need unquote-splicing in the middle of an expression) isn't very nice. So probably a new syntax is warranted.

moonchild2• 5 years ago

Odd, no implementation I tried (abcl, ccl, clisp, cmucl, sbcl) today permits such a form.

waterhouse6• 5 years ago

Hmm. Yeah, you're right. (Come to think of it, when I originally read it, I think I interpreted "what you'd expect" to be "raising an error", or at least I deduced that after trying it myself in SBCL.) Edited my comment.

fiddlerwoaroof5• 5 years ago

I think there might be another issue here: function objects aren't "externalizable" and so you can run into weird errors if you try to compile a form with literal functions.

waterhouse6• 5 years ago

Why aren't they externalizable? You could print a lambda as #<lambda:(args body env)>. The arglist and probably the body would be straightforward (unless the body contains more literal functions, but then you recurse). Printing the env might involve printing complex or even circularly linked structures, but it's no more difficult than pretty-printing objects in general. (Some things, like a pointer to an FFI object corresponding to an open GUI window, would be difficult to serialize, but there are many use cases that don't involve putting those things into the body or env.)

Printing primitive functions could be something like #<primitive:+>, and when an implementation read in a thing like that, it would look in its own set of primitive functions for one named '+ and use that; with a reader macro I imagine you could make that exact syntax work. (In fact, you could print #<primitive:+> on an x86 machine and read it on an ARM machine, and this could be fine.)

I honestly don't see why implementations haven't done this, other than "it was more bothersome than worthwhile".

curist3• 5 years ago

May be unrelated, but that's why I prefer the way JS approach this kind of problem: JS doesn't have macros.

People use callback function to achieve almost the same thing

  function doTexture(texture, fn) {
    beginTextureMode(texture)
    fn()
    endTextureMode()
  }
  
  doTexture(myTexture, () => {
    drawCircle()
    drawRectagle()
  })

At the cost of being more verbose, we get the benefit of one thing less to learn; and simplicity is a powerful feature IMO.

I guess why this is less popular in lispy languages is because

  1. macro
  2. callback functions introduce more indent levels :(

ianthehenry4• 5 years ago

I debated mentioning this explicitly, but decided it was just noise next to the rest of the post. But you should note that, while that macro can be easily replaced with lambdas without much change in ergonomics, there are lots of much more interesting things you can do with macros that do not have equivalent substitutes in a language like JavaScript. (e.g. JSX exists as its own weird pre-processor thing, but you could theoretically just do it with macros instead, and then it could compose with other syntax extensions.)

we get the benefit of one thing less to learn

But learning programming language theory is the best part :)

curist3• 5 years ago

Didn't recognize you're the OP! So my comment may not be that off-topic then :D

But learning programming language theory is the best part :)

That's very true :)

TeMPOraL4• 5 years ago

People use callback function to achieve almost the same thing

There's a QoL difference here with macros - writing out those lambdas can become annoying. That said, the "good style" rule in (Common) Lisp is to prefer lambda-forms in such cases - i.e. cases where the macro parameters are a block of code to be mostly run straight.

In fact, a common pattern for with-macros (of which doTexture would be an example), is the call-with pattern. Example from some random project of mine:

  (defmacro with-logging-conditions (&body forms)
    "Set up a passthrough condition handler that will log all signalled conditions."
    `(call-with-logging-conditions (lambda () ,@forms)))

Which is then used like:

  (with-logging-conditions
    (blah blah)
    (main game code))

All the macro does is, upon expansion, package the code block into a lambda, and passing it to a function call-with-logging-conditions, that does the actual work. So it's like your example, except I don't have to write the lambda myself. This is a trivial case; commonly, macros might accept additional arguments that they process, but eventually they'd still wrap their input body argument in a lambda and expand to a function call with said lambda as argument.

A better use of macros, which you can't replicate in JavaScript[0], would be if you wanted to do something like:

  (do-texture texture
    o O r R x2)

And have it expand - at compile time - to:

  ;; unwind-protect is Lisp's sorta-equivalent of try/finally in other languages.
  (unwind-protect
    (progn
      (begin-texture-mode texture)
      (draw-circle)
      (draw-circle :big)
      (draw-rectangle)
      (draw-rectangle :big)
      (draw-rectangle :big))
    (end-texture-mode))

However silly this looks, this kind of code generation is (one of the main reasons) why you need macros.

[0] - Well, you can if you have a toolchain. Babel is essentially a macro engine for JavaScript, but you can only use it at build time.

curist3• 5 years ago

(do-texture texture o O r R x2)

This might be another reason why I find macros less appealing: macros introduce DSL in form of normal s-expression, but they don't actually behave like a function; macros introduce their own mini-language/syntax.

In the last example you provided, I bet the macro implementation would look like a little interpreter? If that's the case, having a function call like

  doTexture(myTexture, ['op1', 'op0', 'opR'])

and let doTexture handle each cases(ops), might be able to achieve the same behavior, right?

I'm not trying to argue that macros are unnecessary, I really want to like them! Just most of the time, I find functions are sufficient enough.

kamaal• 5 years ago

People have macro use cases more often than they think. I work in devops these days and almost every week I have a situation where I have to generate repeated patterns of code using the language in which Im writing code(Python/Shell/Perl etc).

Many times these are done through what is known as manual work. In my experience if someone is complaining about laborious manual work. Lack of quick automation, they are basically in a scenario where the language lacks features to generate code to do work, which they have to now do instead.

Its for this reason I have to write adhoc perl scripts to generate shell commands, and sometimes even whole shell/python snippets/mini-programs. All because Perl makes it easy to generate text(code). Perl now acts as a macro facility for Shell/Python.

If you pay close attention these patterns will begin to show all up over the place.

nathell• 5 years ago

Clojure alleviates this to a great extent by automatically namespace-qualifying symbols within quasiquoted forms, which prevents clashes between let-bound symbols (which are never qualified) and ones from the global scope.

It’s still possible to shoot yourself in the foot (see https://blog.danieljanus.pl/2020/01/21/middleware/ for my personal story), but for that you need `binding`, `with-redefs` or `alter-var-root`, which are all relatively rarely seen.

mst4• 5 years ago

Time flies by when writing clojure on a train?

mst4• 5 years ago

In case anybody's wondering what the heck I meant here, it's a reference to the song Time Flies By from Half Man Half Biscuit whose lyrics the title of the linked blog post also appears to be referencing.

john-shaffer• 5 years ago

This is a good example of why dynamic vars are dangerous. They are just as perilous as global state, but are more surprising and more difficult to understand. Prefer passing explicit arguments in all cases. You don't have to even think about the behavior, and you'll never spend 8 hours tracking down a missing binding in a commit from 2018.

p4bl0• 5 years ago

Isn't this problem trivially solved by hygienic macros in Scheme?

At least I'm sure it is by fexpr because with them you have complete control over evaluation.

For those who don't know, fexpr are closures (i.e., lexically scoped functions) that are called with unevaluated arguments (just like macros) and are implicitly passed the dynamic environment (i.e., the call site environment). That way you have all the powers of macros and functions with fexpr, and no scoping problem.

But for the capturing problem mentioned in the article, I'm quite sure hygienic macros are enough.

davbryn12• 5 years ago

"So we’re supposed to be writing a game, right? But in order to make progress, we have to fix a bug. And in order to fix the bug, we have to write a test. And in order to write a test, we have to write a test framework. And in order to write a test framework, we have to understand a thing or two about macros."

You could just fix the bug?

Game development is a different architecture to 'regular' system development (especially if it is a solo project).

If you are testing to ensure the bug isn't reintroduced then you haven't fixed the bug.

colejohnson66• 5 years ago

If you are testing to ensure the bug isn't reintroduced then you haven't fixed the bug.

It’s called regression testing, and it’s a (fairly) common thing.

davbryn12• 5 years ago

I know what it is - I don't think its relevant in this project

jbovlaste• 5 years ago

I'm probably late to the discussion, but if anybody finds this looking up macros later, I found the HOPL IV paper Hygienic Macro Technology[1] to be a thorough look at the subject that hardly gets mentioned. It turns out solving these sorts of problems elegantly isn't some technique that the author has missed, it's an active area of research. Newer Lisps compete by having new ways of constructing macros that avoid these problems (say, Racket's syntax-parse[2].)

[1] https://dl.acm.org/doi/10.1145/3386330

[2] https://docs.racket-lang.org/syntax/stxparse.html

TedHerman• 5 years ago

An older Macro, who also had his problems. https://en.wikipedia.org/wiki/Naevius_Sutorius_Macro

The Problem with Macros

Related Stories