Why does Mill use Scala?

https://mill-build.org/mill/depth/why-scala.html

by lihaoyi • 1 year ago

73 75 1 year ago

koito17• 1 year ago

The article fails to mention whether Scala can ensure code is deterministic and hermetic. Starlark code is deterministic and hermetic, but the article never mentions this. Unfortunately, Starlark does not have static typing, but I think types would make Starlark one of the best languages today for build configuration.

In the Clojure community, there was a huge push for "builds are programs". I somewhat agree with this assertion, but I also think "one should restrict the class of programs a build belongs to". Neither Clojure nor Scala, compared to Starlark, seem to offer a way to ensure builds belong to a deterministic subset of programs.

Thus I am still wondering "why Scala?". I have never used Scala, but reading this whole article gives me the impression that Mill is the Scala equivalent of Clojure's tools.build. That is not what I would want in a build system.

fmbb3• 1 year ago

What is special in starlark that makes it hermetic and deterministic?

Scala is deterministic.

If you call functions that have side effects and nondeterministic behavior you can fall outside the comforts of determinism. But you can stumble upon library functions someone wrote in Starlark that accidentally put you there as well.

The Starlark homepage says

Hermetic execution - Execution cannot access the file system, network, system clock. It is safe to execute untrusted code.

But the last time I wrote Starlark it was to define build targets in Basel. And executing the builds definitely accessed my file system and the network, otherwise builds would have no results.

cbeach• 1 year ago

Bazel enforces a hermetic sandbox for Starlark to operate within.

All input files have to be declared, and the build process can only see files that are declared.

Starlark cannot access arbitrary files at runtime, and it deliberately has no APIs for things like system time or random number generation, or global state.

Scala, on the other hand, has no such restrictions. As much as I love Scala, I think it’s an odd choice for a pure, deterministic system. Although perhaps if you use Scala to build a DSL (an area where Scala shines) you could engineer a pure functional sandbox within Scala.

michaelmior• 1 year ago

Scala is deterministic.

I don't see how Scala is any more deterministic than any other language.

thirtyseven• 1 year ago

Bazel splits the build into multiple phases. Starlark only comes into play in the first two, load and analysis. During these phases, Starlark code doesn't have access to the filesystem, except in a few very limited cases like using the glob() function to expand a wildcard to a list of source files. Furthermore, it only generates an abstract graph of build actions. The Bazel engine is responsible for executing this graph in later stages, which might result in non-hermetic things happening but usually not.

Starlark has intentionally limited functionality such as lacking Turing completeness or global variables. This provides guarantees that it can be executed in parallel and will have a finite runtime.

toprerules• 1 year ago

The best config system I've ever seen used plain old Python to generate static configs. Everyone knows Python. Python is easy to do data munging in, as demonstrated by it's popularity as the #1 data science tool. There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause. Even Starlark is just a dialect of Python.

You can spend decades building a complicated configuration language, use a bespoke functional language as Mill does, but if you're a single company that can enforce code quality and just wants to get the job done, I feel like everything else is just unnecessary and over-engineered to scratch some academic itch for a "better system" that enforces "purity" at the cost of velocity.

I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one.

MathMonkeyMan3• 1 year ago

This does work well. A team I was on at a past job did exactly this. On Unix the service literally ran `std::system("python config.py >config.json")` on startup.

The problem with this is that the answer to the question "what kind of configuration can I expect?" is "simulate the script and find out."

If the script is written well, and is short, then the parameters that are filled in by the runtime environment are apparent. Over time, though, there is a risk that the script will not remain written well, and it almost certainly won't remain short.

siriusfeynman• 1 year ago

An approach I use is splitting my config tools into 2 stages

Stage 1 creates a "explicit" config that can be exported to plaintext that contains exactly what is going to be created/modified with no abstraction/simplification

Stage 2 applies the "explicit" config

You get to be as clever as you want in stage 1 to avoid excessive copy pasting or not being able to know what your tool is going to do because all you have to go on is some homegrown DSL

iforgot2210• 1 year ago

You run into the same problem with config DSLs, except now you're dealing with a DSL. Config is almost never going to be static.

threeseed5• 1 year ago

Everyone knows Python

No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc.

But by having a programming language as your build tool you now make it harder for new people to onboard. As in order to build project they often need to some unique, specific to the language syntax. And in order to find this syntax they look around on Github and because it’s a programming language every project has their own unique, specific to the project approach.

Versus something like Cargo.toml where it’s simple and consistent regardless of which project you look at.

wocram• 1 year ago

I think it's hard to argue that Cargo.toml is any simpler than Python. Json might be ubiquitous enough for anyone to read and understand, but if Python is foreign than toml is no better.

emidln• 1 year ago

No they don’t. Just like everyone doesn’t know Cobol, Fortran, Scala etc.

Sure somebody might not have Python experience, but it's pretty easy to just not hire someone who says they don't know Python and isn't willing to learn for the role. I don't know that you'd filter out many candidates out of any random 100 devs.

iforgot2210• 1 year ago

So then they need to know toml (Tom's Obvious Minimal Language)? https://github.com/gtk-rs/examples/blob/master/Cargo.toml I don't know what this file says.

eptcyka• 1 year ago

Ah yes, the age old belief that all software is complex enough that one must first run some other bespoke turing complete program to build every single piece of software.

And of all the languages to pick for this, python, with it's non-hermetic execution environment is bound to bite you in the ass, once your buildscripts start depending on libraries. Oh, you could use poetry to solve the library issue with python, or maybe it'll be setuptools, pip or whatever is the flavour of the month in python packaging.

After fighting with Nix for a sufficiently long time, I think most language specific build tools are not neccesarily the best solution to the problem of automating a build for bit of software written in language X. Complex projects will eventually evolve to depend on multiple languages (unless you're the Linux kernel), at which point the specialized language build tools turn into cumbersome barriers in the build process, where different build tools are not aware of the caching, conventions and configurations of any other tool. As such, in an ideal world, any new language would come with a compiler or bundler that can be supported well by higher level build/packaging tools. And bespoke python scripts ain't that.

aidenn0• 1 year ago

I don't know about Python specifically, but using a language I'm familiar with to generate ninja files (+ any header/environment/&c) for the build has become my go-to way of doing builds in the past 18 months or so.

IshKebab3• 1 year ago

Python is a terrible choice for that sort of thing. Who really wants to have to set up a venv and deal with pip nonsense just to write a config file? Hell even installing Python is sometimes difficult.

MathMonkeyMan3• 1 year ago

Stick to the standard library as of an oldish version of python (3.6?) and it's pretty much zero-install zero-config.

iforgot2210• 1 year ago

If only Python had the equivalent of npm.

have-a-break• 1 year ago

You could say the same thing about setting up a JVM...

fastasucan• 1 year ago

You don’t need any of that now that we have uv (https://github.com/astral-sh/uv)

lmm2• 1 year ago

Scala is hardly some obscure bespoke language. It's a top-20, maybe top-10 programming language, that's been around for 20+ years (and had far fewer breaking changes than Python over that time). Most Python translates directly into Scala, but with the benefit of a proper sound type system and full IDE support. And it's a great language for data munging.

bdangubic• 1 year ago

this “top 10” your LLM hallucinating? :)

makeitdouble• 1 year ago

The claim sounded outlandish, but Scala looks indeed to be around the top 10~20 languages in hiring for instance:

https://www.devjobsscanner.com/blog/top-8-most-demanded-prog...

Scala is only in 0.5% of the scanned job offerings, and is far far behind the major languages in numbers, but I was surprised there's more demand than Rust or even Perl to be honest.

hcarvalhoalves• 1 year ago

I also think that now that LLMs are on the rage, how much context do you think they have for bespoke config language vs Scala vs Python? I think we know the answer to that one.

Nothing against Python, but of all the reasons to choose a technology, whatever is more represented on the dataset of some LLM is the worst reason.

This is a death spiral. There's no hope for the future of this industry if newcomers are thinking like this.

iforgot2210• 1 year ago

I cared about programming languages when I was a newcomer. Stopped caring about 10 years ago. They're just tools, each with their own gotchas and different design choices I couldn't care less about. Between two tools that both work ok, I will definitely pick whichever one my team and I can learn the easiest, and that includes LLM coverage.

asalahli• 1 year ago

There's boundless libraries to make Python more functional, use stricter typing, or reduce the amount of side effects it can cause.

What are some examples of a library that can limit or prevent side effects of a piece of python code? I could use one right now.

agentultra• 1 year ago

I'm not big on programming languages being used for configuration. It adds a lot of complexity and maintenance burden. Configurations are often read and your reader isn't a compiler but now they have to imagine what the final configuration state will be after evaluating the "program" that generates the configuration. I think plain old configuration languages are better, even if verbose, since the usual text-based tooling works quite well for managing, searching, etc.

I use nix a lot and the main thing that bothers me about it is the language. People are quick to get clever with it. It becomes a morass of code that is difficult to read for anyone but experts and when it breaks and your not that expert... good luck fixing it.

winwang2• 1 year ago

Unfortunately, I find that real-world configs don't quite conform to easy understandability past tutorial examples, i.e. k8s and yaml.

vander_elst• 1 year ago

It seems this is mainly a scala project and then they are using scala also for the configuration, it probably makes sense for them.

kunley2• 1 year ago

"Mill is a fast, scalable, multi-language build tool that supports Java, Scala, Kotlin, and Python".

So, while I understand this tool can resonate in the JVM world, I have no idea why one would want to pull Java into their toolset in order to build Python.

wavemode• 1 year ago

What does it even mean to "build" Python?

tacticus• 1 year ago

shouting at pip inconsistencies

pletnes• 1 year ago

I imagine creating a wheel file with your code, metadata and resources? Also recall that a lot of python packages have dependencies written in C, Fortran, Rust etc.

Also, many tools exist to create executable programs - basically bundling the python interpreter with some .py files, etc.

kunley2• 1 year ago

Yeah had the same thought actually

briankelly• 1 year ago

I don’t know who is using mill but I’d guess Scala people and Scala frequently implies Spark which in turn implies PySpark. For instance I could see it being useful for a team building a source connector which will be Scala-centric but need some PySpark additions as well. Even without Spark, Scala is frequently used for data centric applications and those kinds of teams would naturally be working with python as well.

Kwpolska• 1 year ago

Because it’s a Scala project, written by a Scala fan, simple as that. No need to come up with extra justification.

gdgghhhhh• 1 year ago

The title really confused me until I realized this has nothing to do with https://millcomputing.com/ :-)

pabs32• 1 year ago

Wonder when Mill Computing will become more publicly active and have hardware available.

waste_monk• 1 year ago

It seemed quite promising but from the outside momentum appears to be almost completely stalled, with only a handful of posts per year on the forum.

I'd be curious to know if there's progress being made behind the scenes.

rpcope1• 1 year ago

One interesting thing is that there are comparisons to Maven and Gradle but not sbt. Do people just not use sbt anymore or is it omitted because it's also Scala and/or prone to becoming a mess?

videogreg93• 1 year ago

I used sbt in my previous job and didn't hate it. All I want from build systems is to get out of my way and it let us do that pretty well. Very simple to add new tasks as well.

openplatypus• 1 year ago

We use it.

It is actively developed.

It doesn't get in the way.

It does what it says on the tin.

ATMLOTTOBEER• 1 year ago

It’s slow, and the task/setting key resolution makes no sense unless you waste several hours reading sbt docs (time I will never get back). I’m not saying dump sbt for mill but u gotta admit sbt kinda sucks. Afaik this is also the conclusion most larger orgs that use scala come to when they start calling scalac from bazel or similar.

ezst• 1 year ago

https://mill-build.org/mill/comparisons/sbt.html

Lyngbakr• 1 year ago

I'm surprised that Lua wasn't included in the discussion. I'm not saying they necessarily should have chosen to use it, but it's a notable omission given its popularity as a config language.

rubenvanwyk• 1 year ago

A lot of these reasons also apply to Kotlin and Kotlin is arguably simpler than Scala, why not just use Kotlin? It is much more widely adopted than Scala.

ncgl• 1 year ago

The first thing any self respecting python dev is going to do on a new repo is implement typechecking. And the second thing they're going to do is complain about pip/pyenv.

These days I would trade my python experience for scala, even knowing it'd mean less job prospects. We make a lot of excuses for python.

Why does Mill use Scala?

Related Stories