Sweet.js: Hygienic Macros for JavaScript

by philf • 14 years ago

84 19 14 years ago

saurik9• 14 years ago

There seemed to be some confusion during the question and answer segment regarding the relative hygiene of macros in Clojure near the end of the motivation and design talk; while it was totally off-topic for the video (and it thereby made sense to take it offline), I personally wish I had been around afterwards to ask the guy who seemed so adamant that syntax-case was fundamentally better than the Clojure solution (which he claimed didn't do it correctly) why that was the case.

I'm totally willing to believe it, but based on my understanding (which sadly is somewhat limited for Scheme, but fairly in-depth for Clojure) it isn't intuitive to me: it would seem like the way you escape hygiene in Clojure (which by default achieves correct hygiene by attaching namespaces to symbols read for macros or inside of quasi-quote) is quite similar in semantics--but simpler in practice due to being exceedingly less verbose--than using syntax->datum and datum->syntax.

dherman5• 14 years ago

That was me. :) Caveat: I know more about hygienic macros than I do about Clojure, so I'm not in a position to critique Clojure specifically.

Hygiene is (roughly speaking) about getting scope right by default but has never been about forcing it on the programmer. Moreover, there are two components to it, only one of which is easy to achieve in an unhygienic system. It's easy to ensure your macro renames introduced /bindings/ by using gensym. But if your macro introduces /references/ to existing variables, it's very hard to protect against those references getting captured at the site where clients call your macro.

I believe in Clojure they get around this for some cases by letting you fully qualify a reference to a library binding, for example. But what if your macro wants to refer to a variable that's local to it? Such as an unexported library function, or simply a local variable. Again, I don't know if Clojure has an answer to this.

One concrete example: write a `define-inline` macro-defining-macro. At the call site, a user might write

    (define some-local-variable /* something */)
    (define-inline (foo x)
      (+ x some-local-variable))

In an unhygienic system, they should first of all fully-qualify the `+` to be safe (yuck!) whereas a hygienic system just gets that right by default. But more critically, how can they be sure that some client of `foo` won't write:

    (define some-local-variable "something else")
    (foo 42)

Not only will that break, it's not clear how to fix it. A hygienic system gets this right.

I do agree that the state of the art in hygienic macros is too complex. I just haven't seen another system that makes this kind of thing work. But I would like to experiment with Clojure's macros more to see if they have an answer.

Dave

saurik9• 14 years ago

Clojure does not have these issues: when the macro is called, the symbols are already attributed with the full namespace qualification, and usage of quasi-quote inside of the macro definition will also apply namespace qualification to variables local to the definition of the macro; you have to go out of your way to break this. You should spend more time looking into it before claiming to people that it doesn't work correctly; you could easily have just said "that's a good question, we'll look into that after the talk" rather than telling the person that Clojure wasn't as good.

saurik9• 14 years ago

Despite spending a lot of time both in the design Wiki and in the talk discussing the importance of being able to determine whether a / indicates a regular expression literal or a division operator entirely within the lexer (as opposed to using the parser, which is how JavaScript is generally defined), the algorithm that this developer implemented does not actually work.

First off, an example where it works:

    a
    /5/
    7

If you run this through sjs you get:

    a / 5 / 7;

This is because, in JavaScript, statements continue across line boundaries until they are either explicitly terminated by a semicolon or a syntax error, in which case the parse is retried at that point as if a semicolon had been provided. In this case, that means we have a single statement that is a division of these three expressions: a, 5, and 7.

However, let's take a more difficult case:

    a = function() {}
    /5/
    7

This is also a single statement: you are entirely allowed to attempt to divide a function literal by a number, you will simply get the value NaN as output. If you take this file and run it through node, adding a "console.log(a)" to the end, that is in fact what you will get: NaN. However, when first run through sjs, you instead get "[Function]".

The reason is that sjs translated the code to:

    a = function () {
    };
    /5/;
    7;

This is incorrect, and demonstrates how difficult some of these underlying issues are when parsing languages that have intertwined lexer and parser state. :( Attempting some other test cases involving regular expressions (but not semicolon insertion) also failed: it seems a lot more work will need to be done on this before it will be able to process general input (and it is not 100% clear to me that the shortcut required is even possible: I haven't thought enough about it yet to say for certain, however).

(I work on the JavaScript parser for a compile-to-JS language used by people doing jailbroken-iOS development for live introspection of running processes, and thereby that was the first thing I was interested in: how well the parser worked. ;P I have intentions to add reader macros, and then replace all of the extra Objective-C syntax I added with them, but I haven't gotten around to it yet. FWIW: I actually found and fixed a bug in my parser while writing this comment. ;P)

disnet3• 14 years ago

Yeah there are certainly a few bugs remaining in the reader :)

It actually does the right thing if the function is named:

    a = function foo() {}
    /5/
    7

correctly translates to:

    a = function foo() {
    } / 42 / 7;

But clearly I missed the unnamed case. You mentioned finding a few other bugs? Would you mind submitting a bug report on github? I would love to fix those too!

saurik9• 14 years ago

function a() { return /7/; }

saurik9• 14 years ago

Sorry, was distracted by the other conversation with dherman, and really shouldn't be spending any time on this anyway, but I've verified your operator associativity is wrong.

To start with some example code:

    for (var a = 7 in /7/ in 9 in b);

If I run that in node I get:

    TypeError: Cannot use 'in' operator to search for '/7/' in 9

As this code is vaguely equivalent to:

    var a = 7; for (a in ((/7/ in 9) in b));

However, it gets converted by sjs to:

    for (var a = 7 in (/7/ in (9 in b)));

That's obviously different, and gives this error instead:

    ReferenceError: b is not defined

(I really should get back to actually doing my job now, though; if dherman responds again I'll totally notice and follow up: that conversation is really interesting to me.)

SchizoDuckie2• 14 years ago

OK, either your description page doesn't get your point across at all, or you're missing a definition somewhere.

'Wish the function keyword in JavaScript wasn't so long? What if you could define functions with def instead?'

Erh, no? Why in the lord's name would I ? Is that your big selling point? 'I don't like function() because it's too long?'

'Want a better way to make "classy" objects?'

Why would you want to make javascript less like javascript, introduce a dependency to javascript that can read your language, and then compiles back into javascript in realtime, in a way that will obviously make debugging nearly impossible (like coffeescript)?

Am I the only one that doesnt understand the use case of this? Or should this have been presented as just another lexer/parser?

saurik9• 14 years ago

Simple examples are good for people to understand; however, the reason people want macros is usually for much more complex situations where you otherwise end up with a lot of boilerplate or repetitive code: a macro is most simply a way for code to manipulate and generate other code.

Regardless, if you don't like macros (and your arguments are common among people who don't; that said, they are also common arguments against using higher-level languages or even functions), then you won't like this project or any other project attempting to provide similar functionality. ;P

disnet3• 14 years ago

> OK, either your description page doesn't get your point across at all, or you're missing a definition somewhere.

The description page (like the rest of the project at them moment) is definitely a work in process. Sorry about the confusion!

> 'Wish the function keyword in JavaScript wasn't so long? What if you could define functions with def instead?'

> Erh, no? Why in the lord's name would I ? Is that your big selling point? 'I don't like function() because it's too long?'

This is just a really basic example of what you can do. Obviously you could have the same effect with sed but we need something simple to shows what basic macro definitions look like.

> 'Want a better way to make "classy" objects?'

> <provides example that looks like php vomited on Javascript after mating with ruby>

I'm assuming you refer to the macro definition with $ and whatnot. If you have ideas for better syntax please contribute! The notation is by no means final. That said I think it's pretty good for what it needs to do, matching syntax patterns and transforming them into new bits of syntax. Certainly more readable than a full parser/compiler.

> Am I the only one that doesnt understand the use case of this? Or is it just another lexer/parser?

The idea is to provide a middle way between no syntax extensions and writing your own full compile-to-js language languages like CoffeeScript which allow you to add any kind of syntax you want, but at the expense of being able to compose extensions. If you like the class notation of CS you must buy into all the syntax choices of CS whereas macros allow you to add just the syntax you want.

Whether you want new syntax is of course a different question. Other languages like scheme and closjure have found macros useful so sweet.js is an experiment to find out if the same syntactic extensibility that they provide is useful in JavaScript.

Hope that helps!

SchizoDuckie2• 14 years ago

Okay, Now that's a story that makes me understand what this is about, thanks!

How about a more clear description like above on your page with some readups into macro's, and some more complicated examples that tell me as a programmer: "Yes, I dó need macro's!" and I can use them for these and these cases?

Also, don't underestimate how smart your audience is going to be by giving them just 3-line examples. I always find it more clear when there's some actual use cases on the wiki pages. How about a proper class, or a todoMVC style demo, so that people can actually imagine what kind of possibilities this would unlock?

Sweet.js: Hygienic Macros for JavaScript

Related Stories