Typelevel Scala and the future of the Scala ecosystem

http://typelevel.org/blog/2014/09/02/typelevel-scala.html

by LiveTheDream • 12 years ago

135 76 12 years ago

pkinsky5• 12 years ago

I'm conflicted. Type constructor partial application would be great (the current type lambda syntax is just awful), but I really don't want to see Scala fragmented.

mhax• 12 years ago

I think it's fantastic news - even if the net result is just forcing typesafe to address concerns about scala that have been festering for some time.

jghn2• 12 years ago

Agreed on conflicted-ness.

The thing that concerns me the most is that from observation it feels like many of the high muckity mucks of the Scala world would probably be heading in this direction. There's nothing inherently wrong with that, but IMO that increases the chance of serious fragmentation.

robert_tweed• 12 years ago

What's the history behind this fork and what is it likely to mean in terms of fragmentation? In particular what are the irreconcilable beliefs on each side that have led to this?

I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.

pkinsky5• 12 years ago

It's not a niche language. Household names including Twitter, Meetup, Foursquare, and Coursera run on Scala.

willismichael• 12 years ago

According to Cedric Beust, it's still a language that "pretty much hardly anyone uses". http://beust.com/weblog/2014/05/11/more-about-language-popul...

ssmoot• 12 years ago

As someone who's been using Scala for the better part of a couple years:

This seems more like a more ambitious version of the Shapeless and ScalaZ libraries (with some of the same crowd/authors).

I (and I suspect most people who program in Scala for a living) don't use those libraries. Not because they're in any way bad. But there are trade-offs using them IME. The biggest being compile-time. So I get by fine without.

I think this is probably generally good news. In the same way that (I believe) JRuby pushed MRI forward and popularized the case for a multi-threading VM, more advanced type features in this fork probably benefit the Scala community as a whole.

I certainly don't see how it could be harmful. Typesafe isn't going away. The vast majority of businesses will still use their compiler. If you didn't use the advanced-type-based-libraries before, this news probably isn't going to impact you anyway.

So from where I'm standing, having made a late 2.9 to 2.10 transition, and now having made several 2.10 to 2.11 transitions, Scala seems a very mature language/platform to me. Libraries are generally upgraded fairly quickly, and source wise, those releases broke very little (2.11 being the biggest culprit just because of the modularization effort, but that's pretty trivial to resolve).

I dunno. It's not Java. But it's not Ruby either. It's gotten new features since 2.9, but it's broken or changed very little. I find working with it, even working through upgrades, very stable and mostly frustration free.

runT1ME• 12 years ago

As someone involved in the scala community (and scalaz) community, I'd say about 60% of the companies I talk to at the various meetups use at the very least #scalaz. I was pleasantly surprised.

Shamanmuni• 12 years ago

The fork means nothing dramatic. I'll make a short summary of both positions:

Typesafe: we need to offer our clients a good guarantee on stability and compatibility with Java 8 or they won't take Scala seriously. So, for the moment we won't introduce new features or make big changes in the compiler.

Typelevel: we are OK with compatibility and stability, but there are some features that we really want which either don't exist or are a bit quirky with the current compiler. So we will make a fork in which we can add these things while staying binary and merge compatible with yours.

If they make it work, this is great news. There wouldn't be fragmentation as "bleeding edge Scala" would be binary and merge compatible with "stable Scala". And nobody should have hurt feelings, both positions are completely sensible.

VLM• 12 years ago

By analogy its not terribly far off from Debian issuing a "testing" branch off "unstable" a couple years or decade ago.

Its a little more extreme than that, but not much.

Usually forks are more dramatic. This isn't MariaDB or something like that.

For PR reasons they probably should have stolen the Debian "testing" vs "unstable" names instead of getting all forky which is scarier sounding.

lmm8• 12 years ago

What's the history behind this fork and what is it likely to mean in terms of fragmentation? In particular what are the irreconcilable beliefs on each side that have led to this?

What I've managed to put together, corrections welcome:

Typesafe has been moving to stabilize Scala, primarily to make it more attractive to enterprise clients. Prior to 2.10, even point releases broke binary compatibility; now compatibility is maintained for those and they're making efforts to make minor releases binary compatible. The tooling they've produced and the testing framework that builds all the major libraries in the ecosystem and checks for binary compatibility means they now have quite possibly the best binary compatibility regime of any language.

Typesafe recently published a roadmap for the next few releases, saying that the next release would focus exclusively on Java 8 support and remain compatible with the current release. This means work to merge any new features won't even start until January 2016, and those features are unlikely to be in an official release until 2017.

Less frequent and more compatible releases are what a lot of people have been asking for, and it would be very awkward to maintain Java 8 and pre-Java 8 releases of Scala. So the typesafe position that there can be no new features in the release that introduces Java 8 (so that people will definitely be able to switch to Java 8 without making any code changes) makes a certain amount of sense. At the same time it's meant Scala's getting "frozen" in a bit of a weird state; macros were a recently-introduced experimental feature and the official version is kind of half-finished (there's a branch with a more complete version). And there are a couple of features that are quite small and simple but would be a tremendous help for the kind of code the typelevel guys work on (e.g. partial application of types would be a very simple syntactic rewrite, and there's already an implementation as a compiler plugin), and there's long been a feeling that they were coming as soon as we figured out the little details - so to then be told that we won't see these features until 2017 at the earliest was a bit upsetting.

So there was a bit of consternation among the typelevel folks - a group who maintain some quite abstract and powerful libraries that tend to push the type system a bit further than most user code. Some of them started seriously talking about porting their libraries to one of the smaller, more experimental languages with a more elegant type system (Idris, Ermine, or I forget the other option). At the same time, the Scala community now has a lot of expertise in maintaining binary compatibility across quite complex changes, so at some point Miles came up with this idea. I don't think a fork would have happened otherwise - most of the goals can be achieved by compiler plugins (at the cost of being harder to maintain), and if they were going to break compatibility they might as well have jumped all the way to another language. But if the typelevel guys can pull off this "conservative fork", where they'll use their own compiler to compile their libraries but those libraries will then be binary compatible with "vanilla" scala and usable by ordinary scala users, then they get the best of both worlds. It's ambitious, but I hope it works out.

I haven't tried Scala yet, but my gut reaction is I should stick with Clojure until the dust settles. A fragmented community around a relatively niche language is never good.

If they're maintaining binary compatibility then that should keep fragmentation to a minimum. Honestly I don't think this changes much compared to how it was with compiler plugins - if you're building some super-duper-typey project with these advanced features then maybe you'll muck around and use this forked compiler, but if you're a normal user then you can ignore it and carry on using their libraries without noticing any different.

Plus even if the Scala community were to split in two, I think either half would be bigger than the Clojure community.

bad_user3• 12 years ago

Typesafe has been moving to stabilize Scala, primarily to make it more attractive to enterprise clients.

I work primarily for startups and have been using Scala for the last 3 years. Typesafe is entirely in the right to want to deliver a more stable Scala, not just for some mythical enterprise, but for all people using Scala for real-world projects. And this is because people complained about the lack of stability. This isn't just because Typesafe has products to sell, it's also because if you want a language to succeed, then it needs to cater to real world concerns.

Lets compare the situation with Clojure for example - with which you can usually use a library created for Clojure 1.1 and that hasn't been updated in years and that's still usable on top of the latest Clojure version.

Personally I'm happy that this fork happens, because:

     1. if it adds any good features, they'll be merged, 
        and that's what open-source is about

     2. if it fails, then there's no harm done

On the other hand I do not have faith in the ability of the Typelevel people to pull this off. Like every time I tried playing with Scalaz, I ended up with stack-overflow errors - not to diminish their skill or efforts, as yes, they are amongst the very best developers, but these people are too idealistic to address the needs of projects that solve real world problems.

Miles himself speaks with contempt about Play and Akka and their Java compatibility - but other project leaders have a lot to learn from these projects, as they provide a blueprint for how to achieve success - a documentation first effort, reasonable stability, gradual learning curve, a sane deprecation process, mindfulness about the platform they run on and so on. Compare with Lift. Or Scalaz.

kamaal2• 12 years ago

>In particular what are the irreconcilable beliefs on each side that have led to this?

I'm not familiar with this one. But in general most of these forks are generally out of personality clashes where one person just refuses to take another as the boss.

Scala is still not among the top languages used in the industry. This kind of my way or the highway attitude will only fragment the already fragile ecosystem. Imagine any serious software shop wanting to use Scala reading this. If I want to be writing an enterprise application which I plan to keep running for the next decade, should I be choosing a language which isn't just not widely adopted, but the existing ecosystem itself has forks because a few members don't give two hoots about stability(Which is far more important to 99% of the software shops out there, than adding new features).

Instead of doing something like this, and putting your broader community of users in perpetual confusion as to which ecosystem to use, you could rather contribute to something meaningful.

frowaway0016• 12 years ago

Sometimes it's useful to read the blog post before commenting.

dkarapetyan7• 12 years ago

So the elitist community component of Scala that is constantly trying to turn it into Haskell has decided they need an island of their own. Great, they can have it. Now I'll know where to go when I need monadic and monoidal libraries for serving HTTP requests using nothing but type level hacks.

In all seriousness though I don't really know what this is going to accomplish. The article mentions the academic and non-academic components of Scala and it is clear the author falls on the academic side but if that's the case then why not just go and hack on something like Haskell (or Coq), a perfectly unapologetic academic language that has tried to avoid success at all costs. If push comes to shove I'm with team Odersky.

pkinsky5• 12 years ago

Now I'll know where to go when I need monadic and monoidal libraries for serving HTTP requests using nothing but type level hacks.

Monadic libraries with classes like Option, Future, and List? Used in elitist features like for comprehensions? Anyone who writes Scala on a regular basis deals with monads constantly. Why not take 15 minutes and learn how they work?

dkarapetyan7• 12 years ago

Oh no that is all a little too pedestrian for me. I've moved my sights much higher now. These days I use template haskell and applicatives to generate HTML forms: https://github.com/yesodweb/yesod/wiki/Multi-select-field-po....

I'm all for applying theory to enrich the practice of programming but lets be reasonable here. Lists and Options existed way before anyone added monadic sugar on top of it as did mapping and filtering collections. I like category theory as much as the next guy but lets just pull back a little bit. Not everything needs to be point-free, functorial, and shapeless.

saosebastiao• 12 years ago

My biggest frustration with Scala is the exact opposite of their frustration with Typesafe's Scala: it moves too fast. Scala's backwards compatibility problems are obnoxious for the people who don't feel the need to reinvent everything. I can find a version of Foo that works with 2.11 but the only version of Bar I can use only has artifacts for <=2.10.

The play framework exemplifies these problems. Documentation doesn't keep up with releases, blog posts and tutorials don't either. Sometimes I'm looking for example code for some plugin or library, and the code only works with some other version of another dependency that I can't use. I totally understand that people prefer Agile development, but you can't combine Agile development with non-Agile documentation, unless you want to create a framework that only experts can use.

ffreire• 12 years ago

We've been bitten by this as well. We started several projects within a few months of each that also used what we believed to be similar enough versions of Play but there are just enough inconsistencies between versions of play (and Scala language versions as well) to make it exceedingly frustrating to work with.

pea2• 12 years ago

Yeah I've wasted so much time dealing with code for the wrong version of Play. It's horrendously organised: case in point, the JSON parsing (WTF?)

Rezo• 12 years ago

The documentation for JSON parsing in Play is hilarious.

You read the "The Play JSON library Basics" document, it looks like a lot of code for something that should be simple, but you go ahead and implement as explained. After a while, the verbosity starts bugging you, so you read the "JSON Reads/Writes/Format Combinators", followed by "JSON transformers" and finally the "JSON Macro Inception" pages.

The punchline in the last section can be summarized as "forget all that other stuff, just do "implicit val personFmt = Json.format[Person]". I was happy to be able to delete hundreds of lines of code from my project after learning this, but couldn't you just have told me this upfront since it probably covers 99% of all cases? The documentation also has an aura of "Isn't this all very clever?" feeling to it. It's also quite hard to search for anything, as outdated versions of the docs keep popping up in Google as the first results.

noelwelsh2• 12 years ago

You are misunderstanding. The issue you have observed is caused by binary incompatibility between Scala versions. Java has an almost one-to-one mapping to JVM bytecode. Scala, having many features that Java does not, does not have such a direct mapping. Sometimes the way Scala is compiled to JVM bytecode changes, to fix bugs or to improve performance (e.g. compilation of lambdas will change now that Java 8 supports them). This leads to binary incompatibility between Scala versions, though the problem is decreasing with time.

This effort is aiming to keep binary compatibility, so it will have no effect of users of Typesafe's compiler. To quote: "programs which are valid when compiled with the Typesafe Scala compiler would have the same meaning and binary representation when compiled with the fixes in place."

saryant• 12 years ago

Background: I saw Miles' tweet on this last week but didn't follow the apparent brouhaha that followed. All I know of him was his (hilarious) talk at Scala Days in June.

As someone who uses Scala daily, I don't think this is a good thing. I fall squarely into the Typesafe side of the camp—we don't use Scalaz or Shapeless or much of anything from the Typelevel project but Akka and Play are core technologies for us. We don't have a Typesafe subscription, we just use Akka and Play, a lot.

As I observed earlier, there are already several variants of the language in existence and it has been clear for a long time that different sections of the community have different interests.

I don't think the author's examples really hold water. Nobody is using the Dotty compiler commercially, Scala.js is still in its infancy and while I know large organizations have at times run on minor Scala forks (I think Twitter used to but is now on 2.10.4), I'm not sure I believe that this is a major constituency. The official Scala is the Scala.

If this fork gains steam, it's going to be a huge opportunity for FUD around Scala and will hurt its adoptability, especially if existing Scala teams now have to choose sides.

That said, if the Haskell side of Scalaland wants to hang out somewhere else, that might be good too. Too many Scala questions on SO are answered (and upvoted) with obscure Scalaz solutions that make no sense to someone coming in from Ruby or Java.

I like functional programming. I like how monads can simplify things. But I also have to write code that someone without a lot of Scala (or Haskell) experience can maintain.

lmm8• 12 years ago

I like functional programming. I like how monads can simplify things. But I also have to write code that someone without a lot of Scala (or Haskell) experience can maintain.

I agree with this, but I think a same replacement for the type lambda syntax goes a long way towards making those scalaz solutions maintainable.

stuhood• 12 years ago

Twitter has not used a fork of the scala compiler/library since the pre-2.8 days (and maybe not even then, since we have always had a fair fraction of open source code.)

noelwelsh2• 12 years ago

To people worried about fragmentation, retaining binary compatibility means it really doesn't matter. Scalaz can be written in the Typelevel fork and you can use it without issue from code compiled with the standard Scala compiler.

Also, from the announcement:

"As a conservative fork, our aim is to be merge compatible from the Typesafe Scala compiler, and a key objective is to feed pull requests (many of which will resolve long standing Scala bugs or feature requests) back to the Typesafe Scala compiler."

So we aren't looking at radical changes, and hopefully these changes will get into Typesafe's compiler.

So basically it's an opt-in better Scala compiler with no effect on compatibility. What's not to like?

cryptos• 12 years ago

I'm worried about this fork. Scala is far away from being a wildly used industry language. Yes, there are some well known companies using Scala, but how many of them develop a really big part of their software with Scala? Not so much. If you look around you'll see Java, C#, C, C++ (depending on the business) and actually not so much Scala. The ones who think forking Scala does help the community are in a reality distortion field.

Take a look and Linux desktops to see that diversity is not always good, especially if you're in a niche.

propensive5• 12 years ago

I think there is a lot of Scala usage you perhaps don't see. As an example from the corporate world, at least six major international banks (probably more) are doing a significant amount of their development in Scala, some with teams in the hundreds. That's a solid market for Typesafe right there, which isn't going to go away overnight, and won't be stirred by the existence of the Typelevel fork of Scala. Furthermore, they should benefit from its existence because it represents a means for more people to be able to contribute to and enhance the language, without negatively impacting the stable, reliable releases from Typesafe.

benjaminjackman• 12 years ago

I don't understand what features they want to add beyond type lambda's, singleton types, and literals for bytes / shorts.

1. I understand if you spend all day making shapeless / scalaz the lack of sugar for type lambda's is annoying. Why not make a SIP to add some sugar there?

2. Singleton types would have use in ScalaJs as well for encoding. There is already a SIP for this.

3. Byte / Short literals, seem like again, a SIP could be made for the next release.

So all this boils down to getting SIPs into the language faster. So I think the real feature is not having to wait 3+ years for language improvements that significantly improve the QOL of some library writers.

That is a feature I would like as well. I really miss being able to open the mailing list and scan for exciting new features, Instead I know even if someone has a great idea, it'll be years before it sees the light of day. This sucks and I've moved onto to getting my neat idea -> wow I can use this in my language fix to ScalaJS[1].

I think (thought?) about Scala as ycombinator for language features. I see this enterprise mindset coming in now trying to tie down what Scala is and ultimately I fear that it will kill all the things that has made Scala popular in the first place. Pushing to a more infrequent release model might help enterprise customers but it absolutely crushes library developers in the vanguard working to make the Scala of tomorrow by fighting to get the language features of today. They need turn around on the improvements listed above and can't wait 4+ years to get them.

So does Scala take all the strength it has built up from acting as a feature incubator for the last 10 years and switch and lock down what works, it is in a prime position to do so? It has a virtual monopoly on it's space.

Or does Scala continue to act as a language feature incubator giving its coders early access to cutting edge CS idioms in a mainstream language. There are a lot of CS ideas waiting to get their shot.

Some people would see this situation and say Scala is facing a mutually exclusive set of outcomes, it must decide now. Nonsense I say! When confronted with tough choices I think of the famous Yogi Berra quote "When you see a fork in the road, take it."

I ask myself is there some way we can have our cake and eat it too? Can we go on trying out new features while still oferring stability to those willing to wait for it?

What if we moved to model of more frequently releases + less frequent releases? A controlled fork within the ecosystem, similar to what Ubuntu / Fedora does.

Publish 2 major short term releases a year that cutting edge library writers can target and build their libraries against. Vend these are properly tagged releases in maven central. Stick any old SIP in there, the enterprise overlords don't have to know about it. Rapid iteration, with rapid fall-off in support, a fun Scala for those of us that love using new features, bugs and all, for us the language is about growth.

Then have an LTS version released every other year that picks up the features that worked and everyone seems to love and that will be supported for years into the future. Relatively slow iteration with slow fall off in support, vend this Scala to enterprise users with a coherent multi-year support policy that just oozes with that 'this is rock solid and will work for eons to come' professional vibe that makes them happy.

1: I am using ScalaJS and man it feels great, it's like Scala of 6 years ago, I love opening the mailing list to see if there is a new release with some awesome new features / improvements. There is a lot of green territory over there because it's targeting a new platform and that will settle with time, but already it beats the socks off all the other competitors in the statically typed compiles to JS camp, just no one knows it yet.

ddispaltro• 12 years ago

+1 on ScalaJS. It's pretty amazing piece of technology (Thank's team!) using it in combination with David Barri's bindings for ReactJS[1] and Li Haoyi's uPickle[2]. It makes for elegant solutions with a shared codebase.

[1] https://github.com/japgolly/scalajs-react

[2] https://github.com/lihaoyi/upickle

dustingetz• 12 years ago

do you know of any large apps using scalajs-react in production?

eugene_burmako2• 12 years ago

Speaking of byte/short literals, we can already have a syntax for them with string interpolation. In fact, that's what is already implemented in Spire: https://github.com/non/spire/blob/ac12fd02180e39d2d482217319....

bkirwi• 12 years ago

I ask myself is there some way we can have our cake and eat it too? Can we go on trying out new features while still oferring stability to those willing to wait for it?

Honestly, it seems like this is precisely that.

There's a general feeling in the community that the future of the Scala language and toolset is closely tied to the business goals of one specific company, at the expense of other members of the community. As long as the fork stays close to the existing mainline -- and it seems they intend to -- this seems like it should work out well for both sides: the community gets to fix some longstanding pain points with their tools, and Typesafe gets a fertile source of pre-battletested improvements to roll in to their slower-moving distribution.

eeperson• 12 years ago

Don't compiler plugins provide a way to provide a way to experiment with language changes more quickly?

propensive5• 12 years ago

Compiler plugins are powerful, but still too limited. We can't change parsing rules with a compiler plugin, for example. There are also potential compatibility issues with using some set of compiler plugins, and there's a performance penalty of implementing new features with plugins (which, by their very nature, are optional), rather than in the core compiler.

stephen• 12 years ago

How is ScalaJS's output size lately?

sjrd• 12 years ago

170 KB to a few hundred KB for typical applications. That's will full optimizations which take a few seconds to apply.

The fast dev cycle version weighs between 1 and 2.5 MB.

propensive5• 12 years ago

I don't understand what features they want to add beyond type lambda's, singleton types, and literals for bytes / shorts.

We have more ideas on the table beyond these three, but Miles will go into more detail about these in his next blog post.

zak_mc_kracken• 12 years ago

Regardless of the outcome of this initiative (which has little chance of succeeding, but best luck to them), this is pretty bad for the Scala ecosystem overall. There's a general lack of confidence from the community toward Scala and Typesafe and this certainly contributes greatly to the general uncertainty about Scala's chances of success.

frowaway0016• 12 years ago

I guess it was only a matter of time until the "Scala will surely die!!! Really!!! This time it will go down!!!" crowd would show up ...

vorg• 12 years ago

When I see all the various Scala compilers out there (Typesafe Scala compiler, LAMP/EPFL Dotty compiler, Scala.js, Scala Virtualized compiler, Scala.Meta system, and several research and private variants of the Scala compiler, IDEs own Scala variants) compared to the lone version of the Groovy one, I realize just how detrimental to the Groovy ecosystem was Codehaus Groovy's practise of ignoring creator James Strachan's JSR-241 standardization effort and redefining "Groovy" to mean the org.codehaus.groovy implementation only. At times it seemed they were scuttling others attempts to build a variant compiler, and even cannabalizing the annotation-based addon market like when they duplicated the Groovy++ static compilation "speed booster".

_gkossakowski• 12 years ago

I'm somewhat surprised that everybody missed the fact that Scala 2.12 has more to it than just Java 8 compatibility and stability.

We're working on replacing the whole backend in scalac. We'll ship faster and easier to maintain optimizer that fixes lots of long-standing bugs in scalac. In fact, it will allows us to fix the second most voted issue ever: https://issues.scala-lang.org/browse/SI-4633?jql=project%20%...

More details: http://magarciaepfl.github.io/scala/

-- Grzegorz, member of Scala team at Typesafe

Typelevel Scala and the future of the Scala ecosystem

Related Stories