Scala Macros: “Oh God Why?”

http://blog.empathybox.com/post/19126121307/scala-macros-oh-god-why

121 56 14 years ago

blog.empathybox.com

vrotaru• 14 years ago

I seems to be a rather common tension in open source projects developed by research institutions.

The goal of users of those projects is immediate usability which clashes with goals of developers, research.

I have some doubts where one can get PHD (publish an article with high citation index) by improving a compiler error messages. As opposed to adding a new cool feature to the language/compiler, cryptic error messages be damned.

CookWithMe• 14 years ago

True.

I still think it is a shame that neither Red Hat nor Jet Brains has joined forces with Scala/Typesafe. Both of these companies know how to ship products and make developers happy. This is something that Typesafe has to learn now. On the other hand, Red Hat and Jet Brains need to learn how to do language research... (which may actually be harder, once they go beyond fixing the obvious Java flaws).

cageface4• 14 years ago

As much as I personally like Scala I understand why a lot of people are taking a wait & see attitude. I think it's not too unreasonable to imagine that a less ambitious Java++ like Kotlin might find the traction that has so far eluded Scala.

rbehrends3• 14 years ago

With respect to publishing, it depends on what conferences you target with your publication. For example, at the intersection of the software engineering and PLDI communities, you'll have quite a few researchers interested in proper tooling. ASE (Automated Software Engineering), for example, is a highly regarded conference that is really interested in this type of research.

There are also other incentives for researchers to create usable tools and implementations (assuming that the tools are actually meant to be more than experimental prototypes).

To begin with, compiler developers traditionally "dogfood" their compilers. They may still be willing to live with things that you wouldn't have in a commercial product, but they can't and don't completely neglect usability, either.

Second, usable tools are more likely to be adopted by other researchers, creating a positive feedback loop with respect to future publishing opportunities.

chalst• 14 years ago

Macros make language design experimentation easier. This issue is not orthogonal to tackling the other issues the OP cares about.

There are good reasons to doubt the overall value of macros in production code - I won't argue this either way - but OCaml has had success by separating the macro expansion mechanism (camlp4) from the main language. People who want macros can have them, people who want a production language without macros can have that too.

baconserker2• 14 years ago

The author doesn't even know what macros are: "I understand macros to provide two things (1) forced code in-lining and (2) a kind of non-strict semantics or lazy evaluation of arguments"

smanek• 14 years ago

What else do you think macros provide?

As a former professional Common Lisp dev, those two features strike me as the real difference between a macro and function call.

rbehrends3• 14 years ago

Scala already has @inline and call-by-name, so it doesn't need macros for either purpose.

Macros in Scala are primarily of interest for metaprogramming; it's a fairly natural direction for Scala to take, given that Scala has always positioned itself as a language that supports internal DSLs.

ghusbands2• 14 years ago

For Common Lisp, think also of reader macros.

Similarly, in syntactically richer languages, macros could add syntactic constructs that are not akin to function application. With sufficiently powerful macros, you can add, for example, new infix operators, a new form of switch/case or a new way to declare variables.

noelwelsh2• 14 years ago

Macros provide new binding constructs. That is their number one use in my 10+ years of Scheme programming.

mindslight2• 14 years ago

The other difference is (of course) what the macro returns - more code to be compiled, which enables things like binding of new symbols/types/etc. The most important part is that the generated code is adjacent to user code, so the two are lexically coupled. But in CL land, you probably take the codegen for granted and the lexical coupling doesn't matter as much.

Although, Scala specifically, I ran away from. The language itself isn't terrible, it just forces one to confront the endgame of the bankrupt Java philosophy. The stdlib is basically the cross product of pre-optimized category theory with noun-hell, and idiomatic scala is far too overabstracted through abuse of first-class modules (which, admittedly, was a design goal).

rbehrends3• 14 years ago

The post raises a few good points -- those are real problems in using Scala -- but the problems are mostly a result of the compiler targeting the JVM as its primary backend. Obviously, a JVM-based language can leverage a rich infrastructure of libraries, but it also will have to deal with the fact that the JVM is primarily designed for Java, not as a more general backend (such as the LLVM or the CLR).

(That the compiler is slow has other reasons, but is also something that they've been working on simultaneously for 2.10, so it's not as though that has been pushed aside by macros.)

lysium• 14 years ago

The listed points are valid, but they are by far not as easy as the author thinks they are. I don't say they are not improvable, but they are not 'little things' that need to get fixed before new features come in. For example, the jars are large because every closure needs its own class (thank you jvm!). Compile time is high because of type inference. Error messages and stack traces are cryptic, because you do high-level programming but get low-level diagnostics. etc.

baconserker2• 14 years ago

For those curious about what Scala macros actually are: "This facility allows programmers to write macro defs: functions that are transparently loaded by the compiler and executed during compilation. This realizes the notion of compile-time metaprogramming for Scala." http://bit.ly/zeMjOb

http://scalamacros.org/usecases/index.html

soc882• 14 years ago

I think many people share some concerns about macros, especially the need for good error messages. A far as I know Scala's language designers have been against including macros for a long time. So the proposal for macros has to be damn good if they change their opinion.

Macros are supposed to be a replacement for writing compiler plugins in many cases (compare with Java annotation processing vs. Java compiler plugins). The API is easier and more straightforward, semantics are easier to understand and don't require separate setup step.

In the meantime work has of course not stopped:

- Syntax has been simplified and several confusing bits have been removed/deprecated (Octal literals, FP literals without digit after dot, ...).

- The jar size of the library has been reduced, while adding fixes and functionality.

- Value classes have been integrated, which provide the performance of primitives with user-defined classes.

- Inlining has been improved a lot.

- Compilation speed has been improved a lot.

- Some collection classes have been performance optimized and are now a lot faster (and sometimes faster than the Java implementation).

- Play 2.0 and Akka 2.0 have been released, both with huge amounts of documentation.

- Documentation has been improved.

- ...

My personal impression is that while Scala has roots in academia, solving real-world issues is the focus of the team.

thebluesky• 14 years ago

I look forward to improved docs in the standard library when that happens.

anonymoushn2• 14 years ago

As far as I can tell, value classes provide the performance of primitives with primitives. Is there a way to make a value class that contains more than one primitive, so that you could have semantics similar to those of a struct in C?

billjings2• 14 years ago

I'd like to single out this line from the linked article:

"Unlike Java, type signatures in Scala don’t really explain idiomatic usage to mortals very well."

This is a great diagnosis of one of the big issues with Scala's style of static typing. Everyone knows that the type descriptions take a bit to grok compared to a C-style languages. Why? Well, in C-style languages, a function definition is itself a representation of idiomatic usage. In Scala and its brethren, you have to translate from one to the other.

I'm sure that's a commonplace observation, but it had not jumped out at me like that before.

sausagefeet2• 14 years ago

Ii have not used Scala but I have found the opposite to be true in Ocaml and Haskell. The type definition of a function is quite often sufficient information to infer the usage. I think this is because of the type variables. Looking at the definition of fold in Ocaml, for example, it becomes very obvious what is the input output init and the signature of your function from the type where such information generally seems to be less clear in C-like languages.

soc882• 14 years ago

The same is pretty much true with Scala.

The only exception where the signatures are really long and complex are parts of the collection infrastructure, because Scala developers tried to make it work as intuitively for users as possible: E.g. making non-collection classes support collection operations, always returning the most "precise" type, etc. etc, which is very hard and painful work not done in any other language to this day.

In the end while I hope signatures get shorter and more like the use case signatures already existing in documentation, I don't think it matters much.

Untyped languages have more or less no useful signature at all and they are still pretty much alive.

billjings2• 14 years ago

The point is not whether you have sufficient information to infer the usage. Do you have to infer the idiom at all? Usage is homomorphic with declaration in C, so this step is unnecessary.

hp2• 14 years ago

The bug in this post is thinking that if you have someone willing to contribute feature X to an open source project, they were also available to contribute feature Y.

Open source patches are all itch-scratchings. For Scala macros, it's a researcher at EPFL, I believe. I don't think you could write a thesis about fixing bugs in the IDE. So this contributor was not available to submit IDE bugfixes; they _were_ available to submit macros.

It's the same for other contributors. Someone building an app on Scala may be available to submit a fix for some scalability issue they are hitting in production; they will not be available to work on macros, or on IDE bugfixes.

Open source projects get the fixes they get. You can't act like the priorities are set by some central product manager.

So I think it's just not correct to argue that "the priorities are wrong, why are they doing this instead of that, etc." - everyone is doing what matters to them. Some people _are_ working on IDE bugfixes, other people are working on macros. People can do what they want.

cageface4• 14 years ago

Sure, but the communities behind all the FP languages seem to share this same weakness. For obvious reasons, they attract people much more interested in hacking on funky compiler techniques than on mundane things like I/O libraries and documentation.

In contrast, I think Python and Ruby both owe a lot of their success to the people willing to put a lot of energy and polish into this essential but less glamorous work.

hp2• 14 years ago

The Scala community is more of a blend on this front. It has FP aficionados, but it's also in production use and has commercial contributors focusing on practical issues. If Scala continues to gain popularity, it will probably continue to get more of a practical focus (there are only so many FP geeks in the world).

The other thing with Scala is that it compiles to bytecode; once compiled, it's just Java. And you can always use any Java library. So there's that base of a very solid JVM runtime and very solid ecosystem of libraries that's always there.

tikhonj• 14 years ago

I don't know, this has not been my experience with Haskell at all. Not only do plenty of people work on all sorts of libraries (there are some really great web frameworks, for example), but they also work on all sorts of "less interesting" stuff. For example, there are some people working to improve Hackage, which is the site that contains Haskell's libraries and their documentation, and Haskell has really good support in Emacs (which is much better than good IDE support!).

eli_gottlieb• 14 years ago

If you want the mundane stuff done, do it yourself or pay someone to do it for you.

sreque3• 14 years ago

I want to emphasize your post with even more information. I believe the person implementing macros is actually a first-year PhD student at EPFL and he decided to do his first major project on porting a Nemerle-like macro system to Scala (http://www.scala-lang.org/node/10978). Of course he isn't going to do a project on making Scala's documentation as rock solid as PHP's!

Also, as for why macros are being put in the language: this is what Martin Odersky had to say on the subject (see http://www.scala-lang.org/node/12494):

"The intention of macros, and of Scala's language design in general, is to simplify things. We have already managed to replace code lifting by macros, and hopefully other features will follow.... Optimizations such as on Range.foreach are another use case. I believe that in the long run macros can be a simplifying factor."

Macros therefore can actually satisfy at least one of the complaints the Yammer CEO made about the language: slow foreach loops, along with simplifying the language in general.

bkudria• 14 years ago

Minor correction: Coda isn't our CEO.

fadzlan• 14 years ago

You are right. But I think the point that the author makes is that pure(or almost pure) open source project will have only "interesting" contribution, where as by far what makes a programming language usable are those "unglamorous" fundamental things.

Yes, whoever contributes can choose whatever they wanted to contribute. But most successful project, companies or startup or anything else for that matter, involves doing things that most people would rather do, but necessary.

When you choose a programming language for your business, your business/life or anything related to it will depends on the language. I think its fair for anyone to choose a language to make sure those necessary boring stuff are robust in the language.

Those who contribute can choose what to contribute. Those who consume, can choose what they want to consume. Probably all of us have been on both side of the fence, so I would say its a fair assessment.

noelwelsh2• 14 years ago

In this case the OP's point is moot because every single one of his gripes is being addressed by Typesafe, the company that is backing Scala. They have a team working on the Eclipse plugin. They have recently released a new documentation site. They are addressing binary compatibility with the Migration Manager. etc.

boredandroid• 14 years ago

This would make sense if it were just a question of getting all the feature boxes checked off and implemented. The fact is though, in language design each feature tends to complicate everything else. For example, I suspect good error messages, debuggability, and IDE support are all set back by this.

sreque3• 14 years ago

Read my post in the same reply level. The creator of the language believes macros will actually make the language simpler. This is made possible by the fact that other language can be re-described and re-implemented in terms of macros, possibly even turning some language features into libraries. In his interview with InfoQ recently, we learn that his long-term goal is to decrease the size of the language in the future if possible. Directly from http://www.infoq.com/articles/odersky-scala-interview:

InfoQ: There has been some debate in the Scala community about the need to add features versus keeping it lean. Can you give us a sense of where Scala is headed? Does Scala need to become a bloated multiparadigm and multipurpose language to be considered mainstream?

Martin: I think the consensus is very much that we keep it lean. Scala is a compact language despite its enormous spread of use cases because it has always tried hard to unify things. The same concepts describe objects and functions and components. Instead of adding features, Scala has very powerful abstraction capabilities that let users define similar features themselves in libraries. In the future my ambition is to make Scala an smaller language, not a larger one. I know it is hard, but we will try.

eloisant• 14 years ago

Macros are useful in Scala. Mostly not for application developers, but for library or framework developers. It enables you to do things while keeping the typesafety nature that makes Scala so great.

In Play, we have generated code for forms validation and json serialization. This enables the compiler to see code that can't work, while a more dynamic approach would lead to an exception at runtime.

stephen• 14 years ago

Where I work (i.e. write code for money, to satisfy Ted Nyman's requirement that my opinion matter), our reaction was exactly the opposite--we're very much looking forward to having macros in Scala.

If this was a more esoteric language feature, I'd be inclined to agree, but macros (if done well), I think will be a huge boon to the language.

I do all sorts of hackery in Java now (external code generation, heavily using annotation processing tools (APTs) which are a very, very limited form for language-based code generation) to do semi-metaprogramming stuff and can't wait to do the real thing.

(And do it well, e.g. not to make things "even more cryptic" just for the fun of it, but to solve real problems.)

draegtun• 14 years ago

For those interested the next release of Rakudo will be the first to (partially) implement Perl6 macros.

ref: http://strangelyconsistent.org/blog/macros-progress-report-d...

odersky• 14 years ago

The blog post and tweets seem to say the Scala community only cares about fancy language stuff and not about the developer experience. Nothing could be further from the truth. Take Typesafe for example. We have three full time engineers on staff to improve the Scala IDE for Eclipse. We made a lot of progress and are continuing to do so. Nobody pays us for any of that; we do it because we know that IDE experience is crucial for Scala developers. Typesafe is just one company and it cannot pour unlimited resources into all aspects of Scala development, so we need to rely on the community for that. And the community does step up to the task. One of the criticism was on documentation. I wonder whether people have recently looked at http://docs.scala-lang.org/ ? In my mind, that's community-driven development at its best!

Why macros? I was a long-time sceptic. I now tend to think about them differently because I believe we hit on a brilliantly simple scheme that can express a lot of different use-cases. In particular we'll be able to do the analogue of Microsoft's LINQ with macros. And we can express optimizations such as turning foreach applications into while loops. And we can remove a lot of compiler plugins. And finally it looks like we can remove down the road several special cases in the language and the compiler. So macros might pay for their own complexity handily. But I should also say that at present this is a SIP, a Scala Improvement _Proposal_. It's not yet accepted. And part of the current proposal is also that macros will be enabled only under a special compiler flag.

igor-petruk• 14 years ago

Compiler flag would make code part of the projects dependent on this flag and thus make compilation process more complex in terms of choices how you compile. Developers may end up just enable this flag always, which kills the purpose of the flag. I sounds more reasonable just keeping compilation to scalac *.scala

stuhood• 14 years ago

We definitely appreciate the work that you and Typesafe are doing: thanks! I apologize for my brusque reaction in that tweet.

Regarding macros: it's quite clear that they simplify the job of the compiler writer: you can provide more features, more efficiently to the end user. But every feature has a cost, and increasing the end user's surface area onto the language by providing 1) macros, 2) new features enabled by macros, is very much a double edged sword. Being enabled by a compiler flag definitely helps to minimize end-user exposure to the former.

Macros in lisps can be used as a replacement for many of the features that Scala provides, including call-by-name parameters, (certain types of) implicits, and many uses of generics. If we could deprecate some of those features in Scala by replacing them with macros, then this would seem like more of a net win to me. But that doesn't seem likely: instead we will probably have all N, and Scala programmers will need to decide which of the growing list of ways to accomplish cat-skinning is best.

modersky• 14 years ago

I am very much aware of the double-edge sword nature of macros, and therefore follow and aid their progress with much trepidation. The single thing that sold be was that macros by themselves already could replace a compiler phase, which did code reification (i.e. generate ASTs at run-time; something that's needed for LINQ like technologies). So in that sense, macros, if they arrive, would already have paid for themselves. Concerning possible misuse, I believe the best thing we can do is give an option to developers. So we plan to enable macro definitions only if a special import is present. Something like:

import language.macros

Of course that will not prevent misuse, but it will make it much easier to enforce policies, if someone desires that.

sreque3• 14 years ago

It's been shown formally that call-by-name semantics and call-by-value semantics cannot be expressed via the other using macros (see http://www.ccs.neu.edu/scheme/pubs/scp91-felleisen.ps.gz).

Also, macros cannot be used in Scheme to implement implicits because implicits, like type classes, rely on static type information. I have actually heard of an attempt to implement type classes in Scheme and it required manual type annotations in all use cases. And, while it hasn't been formally proven, intuitively implicits and typeclasses feel like the kind of languages features that would be difficult, if not impossible, to implement with classic macros.

I feel like you have some pre-conceived notions, both about what macros can and can't do, and about what affect they will have on the Scala language when added. However, as others more knowledgeable than me have already pointed out, macros have already managed to simplify the language as is, and their full potential has yet to be untapped. I would suggest giving them a chance.

retronym• 14 years ago

And macros aren't just for optimizations -- they can also provide additional compile time checks, for example checking that regex strings are well-formed, or that the arguments to printf, or printf-style interpolated strings, match the provided format specifiers.

emolitor• 14 years ago

Many parallels to the negative feedback AOP received when it first appeared on developer horizons. It took a while for people to see where they did and did not fit logically into their problem solving toolkit. I can't predicate the future but Macro's fit into a similar bucket. I don't see them as a tool I would apply everyday but on the occasion I needed them the alternatives are blunt instruments.

I'm certainly not an expert in this area and have only toyed around with the SIP and Retronym's examples so take everything I say with a huge grain of salt. But I do hope everyone critical of the SIP and the concept has at least done the same.

dimeni• 14 years ago

I can't wait for scala's macros! Every time expressiveness, conciseness, and performance converge it is a thing of beauty.

denka• 14 years ago

Mr. Gosling attributes many strengths of Java to the fact that while designing it he and his team were working with technical writers, so that language, platform, and documentation were developed in parallel. Additionally, he attributes the need to explain stuff to writers as an important mechanism to influence the quality of the design itself. I wonder if this positive learning was taken benefit of with regards to Scala. Community-driven is great in principle, but after-the-fact, afterthought appearance that this approach creates is unfortunate.

athiguna• 14 years ago

You guys really think that code that looks like this - https://github.com/twitter/scalding/blob/master/src/main/sca... is more readable and maintainable than just Java?

lihaoyi• 14 years ago

"I understand macros to provide two things (1) forced code in-lining and (2) a kind of non-strict semantics or lazy evaluation of arguments"

False. There isn't much else to say, but if this is all he thinks macros are about, he doesn't know anything about them, their uses, or their potentials.

Macros would basically take a large amount of nasty extra-lingual stuff (code-generation, byte-code generation, compiler plugins) and codify it into a systematic, easier to use and easier to understand framework.

F#-style Type Providers? Compile-time typechecked regex literals? Compile-time typechecked html? Compile-time typechecked CSS? This sort of thing is already being done (see Play!) and is pure awesome as a user (i <3 the type-checked HTML templates) but the implementation involves a huge amount of mysterious extra compilation steps and magic that macros could help simplify and codify.

Scala Macros: “Oh God Why?”

Related Stories