Taking ML to production with Rust

https://www.lpalmieri.com/posts/2019-12-01-taking-ml-to-production-with-rust-a-25x-speedup/

195 99 7 years ago

xiaodai2• 7 years ago

Rust will never work except in niche areas because it doesn't allow for easy experimentation and does not provide an iterative workflow like Python, R, or Julia.

I think the future is Julia, as it's as easy to write as Python, is interactive, and can be compiled for fast speeds!

It will just takes time for Julia to have as many libraries, but at some point the tipping point will be reached, and Julia will just eat Python's lunch.

Python was dying before the latest wave of data science and machine learning craze saved it anyway. There are pretty good reasons to believe Python (and Ruby) will slowly wither away. I am careful not to savy that they will die out, because they won't as some computing problems do not require so much speed and hence Python and the like can still pass mustard. But in a post-Moore's-law-world, data is getting bigger, the demand for computing is larger, but programs aren't getting faster, and there is a limit to how much you can optimise Python. Eventually, a crop of speedy and easy to write programming languages will rise up and eat Python's lunch. The early signs are there with tools like Julia.

Ygg2• 7 years ago

Rust will never work except in niche because

It warms my heart to see people don't read TFA. Like at all.

From TFA:

But it was important to clean the air of any possible misunderstanding before getting started. I don’t believe Rust will replace Python as the language of choice for ML - it just isn’t happening, it’s not a thing, neither today or tomorrow.

hu3• 7 years ago

What's wrong in expanding on a topic that was barely mentioned in the article?

Your parent wrote detailed reasoning that's not present in the article.

Also this type of uncharitable comment is against Hacker News guidelines: "Please don't comment on whether someone read an article".

http://news.ycombinator.com/newsguidelines.html

MayeulC• 7 years ago

That may be true in data science, but the tradeoffs are not necessarily the same in other fields. I do not see Python fading away in mine, for instance. If anything, it looks more and more encroached, and even if the Python code becomes legacy code, interfaces will have to be maintained. I still use TCL quite often in design automation. Python is and will still be used to script Blender, Kodi, Inkscape, Klayout, sigrok, and countless others, for instance.

But your week-end project you need to write a bit of code for and want to practice a new language with/just want to code as quick as possible? Sure, the languages will come and go.

Likewise with Rust. Maybe not for iterative design phases, but I can certainly see it being used in production, especially if robustness/speed are needed.

mirekrusin• 7 years ago

Yes, Julia makes some impressive stunts - supercomputing [0], ml where code is very close to mathematical description [1], code reuse [2] etc.

[0] https://juliacomputing.com/blog/2019/04/12/Supercomputing-ju... [1] https://www.youtube.com/watch?time_continue=188&v=9KBaRS2gy-... [2] https://www.youtube.com/watch?v=kc9HwsxE1OY&t=130s

pjmlp14• 7 years ago

I am also betting on Julia.

ilteris• 7 years ago

What do you think about Swift

Certhas3• 7 years ago

I like Julia, but let's be real. We are a long way away from Julia being anywhere near Python when it comes to usability. Error messages, code structure and readability and documentation are afterthoughts rather than a central focus of the community.

Because the core is so brilliant at what it does I am sure that Julia will get there eventually, but it will be a painful way.

For crying out loud, we still don't have a sory for people to save their data...

wtetzner• 7 years ago

For crying out loud, we still don't have a sory for people to save their data...

Can you elaborate on what you mean by this? Does Julia not support writing files?

cwyers3• 7 years ago

It's not like Python's community is done writing libraries and improving libraries. If there is going to be a tipping point it's going to have to start happening before the Julia package ecosystem catches up to Python's.

psv13• 7 years ago

Rust can never replace Python as a front-end because it's compiled and not all that flexible or user-friendly.

Rust can't currently replace C or C++ at the back-end because it doesn't have good enough support for GPU computing yet.

excerionsforte• 7 years ago

"The Rust ecosystem is indeed rich in ML crates - just take a look at what a quick search for machine learning on crates.io returns. No need to go and rewrite everything from scratch: I picture linfa as a meta-package, a collection of curated algorithm implementations from the Rust ecosystem. The first stop for your ML needs, as scikit-learn for Python."

Rust doesn't have to replace anything. It can be used to create complementary packages especially ones that use C/C++. The point of introducing Rust into your ecosystem is for safety + performance which is hard to achieve without discipline.

psv13• 7 years ago

Rust doesn't have to replace anything. It can be used to create complementary packages especially ones that use C/C++.

I'm not sure that I understand correctly - are you suggesting that people use Rust as an additional layer between Python and C++?

pjmlp14• 7 years ago

Right now, NVidia is designing their GPGPUs to run C++ code.

https://www.youtube.com/watch?v=86seb-iZCnI

https://www.youtube.com/watch?v=VogqOscJYvk

With Intel just releasing oneAPI (aka Data Parallel C++) last month, and Khronos pushing SYCL as alternative to CUDA C++.

It took 30 years for C++ to reach this stage, and in some domains (like embedded) it is still fighting for relevance.

That is something to keep in mind whenever to advocate Rust as replacement for Domain XYZ.

saagarjha• 7 years ago

What’s wrong with being compiled?

username90• 7 years ago

In python I can write code one line at a time and they are executed immediately. This means that I can write code to print the data if I want, or try different things etc without ever rerunning the program from scratch which can be very slow for training data.

wdroz• 7 years ago

The 25x speedup is because the gRPC python implementation is very slow [0].

[0] -- https://performance-dot-grpc-testing.appspot.com/explore?das...

kbumsik• 7 years ago

Yeah, by skimming the article, it sound like making gRPC microservices 25x faster, not ML.

nevi-me• 7 years ago

gRPC would account for very little time relative to the k-means algorithm. I think what results in the large speedup is the "embarrassing" parallelism that an async Rust server (gRPC in this case) would give. Knowing the CPU time during the tests could confirm this. I'll try run the benchmarks on my machine this week to verify my assumption.

amelius• 7 years ago

Of course the actual ML is not going to be faster in Rust, because the main tensor operations as used from Python are already written close to the metal. The cost of these operations is 99% of what it takes to run ML. Optimizing the remaining 1% is just not going to have a noticeable effect.

captain_price7• 7 years ago

I was a little surprised to see that instead of directly benchmarking prediction performance on a local machine, author chose to go through that complicated web-server route. I saw author saying this is nothing more than a remote RFC, still, a benchmark should try to remove unrelated factors as much as possible. I would be interested to see how the results compare when this web-service layer is removed.

cwyers3• 7 years ago

Serving model predictions via an API is a pretty common usecase for putting data science models in production. If Python's web service layer is slower than Rust's, that's a real concern even if it's not directly related to model prediction.

sandGorgon3• 7 years ago

Although I am a python guy, if I had to bet on an alternative language here - its going to be Swift.

The amount of evangelism money that Google is spending on Tensorflow is staggering. Its not just t-shirts, but an entire ecosystem - TensorBoard.dev, Tensorflow Privacy, TF Enterprise, Explainable AI Beta, free TPU on Google Colab etc.

And all of them will soon be backed using Tensorflow Swift (https://www.tensorflow.org/swift). This is MNIST for Swift Tensorflow - https://github.com/tensorflow/swift-models/blob/master/Datas... . Mind you this is runnable on Google Colab using TPU hardware acceleration in swift.

Swift vs Rust is a very political thing...but previous discussions on HN have been tilted in favor of Swift [1]

And we havent even touched on the elephant in the Room - Swift may soon hook into OSX Metal (which is arguably the only way to do GPU AI on macbooks now that the nvidia-Apple CUDA divorce is official [2] ). You can already call Metal from Swift...but not write shaders directly IMHO.

[1] https://cloud.google.com/explainable-ai/

[2] https://gizmodo.com/apple-and-nvidia-are-over-1840015246

Roritharr• 7 years ago

This is really new to me, I always thought of Swift as the iOS thing to look into when doing something there.

Very interesting that Google is pushing Swift for their Tensorflow Platform. Any ideas why Swift is of interest for Google?

jorgemf2• 7 years ago

They hired one of the Swift leaders for the tensor flow team

bpicolo2• 7 years ago

Swift playgrounds seem very much aligned with notebook style ML development. They're in many ways a much more powerful version of notebooks.

https://www.apple.com/swift/playgrounds/

Abishek_Muthian2• 7 years ago

You can read why Tensorflow team chose Swift here[1].

[1]https://github.com/tensorflow/swift/blob/master/docs/WhySwif...

k__2• 7 years ago

I had the feeling Swift was much higher level than Rust and it also didn't bring much new to the table.

hellofunk3• 7 years ago

I think the OP is making that point that Python was successful because of how relatively simple and high-level it is, thus Swift has potential in that area for the same reason. Rust is hardly approachable by "the masses", certainly all the ML-first engineers who are not as hard-core on the programming mindshare, hence why Python became king.

throwlaplace5• 7 years ago

And all of them will soon be backed using Tensorflow Swift

Lol at this. It's ~5 years before S4TF has feature parity with TF let alone pytorch. I watch their design meetings almost every Friday (they're public) and not only is the core autodiff runtime nowhere near 1.0 (just a couple of weeks ago they were talking about replacing their bespoke system with XLA) there's almost no differentiable function type library (nn.functional). Not to mention ecosystem (there are probably, at most, 20 people, including the S4TF team, consistently using S4TF). This is not to say that they're not doing great work but that's it's far off.

Edit: I forgot to also say they're going through the Swift evolution process (since they're building the autodiff into the Swift compiler) and it's completely unknown whether it'll be accepted. I have no idea what will happen to the project if it doesn't but if they end up having to maintain their own Swift runtime then S4TF is dead in the water because no one will be comfortable deploying two runtimes and even then no one will trust Google to maintain.

justicezyx• 7 years ago

How can I audit the Friday design meeting?

asparagui2• 7 years ago

As one of those 20 people, I feel like I'm in good company!

Five years is pessimistic. The core library is still in a state of flux, but the team is getting the fundamentals hammered out. Once the foundation is in place, adding higher level apis should be relatively straightforward.

Re: XLA/MLIR, s4tf is actively trying to coordinate with other groups within google. Whatever short term losses there are in time/energy will be paid off down the road when the project can piggyback off other people's progress.

Finally, while the swift evolution process isn't going as quickly as would be desired, the changes needed to merge the tensorflow branch have been identified and are being worked on. It's just a matter of time till the two projects are in sync.

jorgemf2• 7 years ago

I whish it was kotlin instead of Swift. But as much as I want to move away from python, there are reasons to stick with it (ML libraries and low entry barrier for non programers)

mark_l_watson• 7 years ago

When first announced, I was very upbeat on S4TF and the XCode and Swift drops with TF and some examples more or less worked and were interesting.

Since the first early release, which largely worked for me, subsequent releases have generally not worked for me. I even had much more trouble with S4TF than, for example, the Haskell TF bindings that always take a bit of setup since I don’t use the provided docker setup.

A little off topic, but, the Julia Flux ML library is worth looking at if you wanna “turtles all the way down” setup.

rob74• 7 years ago

While Swift may be the "latest and greatest" language with official support, if you look at https://www.tensorflow.org/api_docs, it's only one of several (the others being C++, Java, JS, Go and of course Python). Plus, there are community-supported bindings for C#, Haskell, Julia, Ruby, Rust and Scala). Just to make the picture complete...

throwlaplace5• 7 years ago

Swift for tensor flow aren't bindings - it's a ground up rearchitecting of tensorflow

pjmlp14• 7 years ago

Microsoft also provides Tensorflow bindinds via ML.NET.

So any language that can target .NET is also able to plug into it, and I certainly would pick F# over Swift.

cwyers3• 7 years ago

And R has Tensorflow bindings too. https://tensorflow.rstudio.com/

eeZah7Ux• 7 years ago

The amount of evangelism money that Google is spending

Same for Rust, Swift, Go... it's like we are in another bubble like the dot-com and everything is hype driven.

microcolonel• 7 years ago

Well... not everything, Rust is actually good. Just sit back enjoy the effects of the investment in this stuff, and don't worry about other people being foolish with their money.

kibwen• 7 years ago

Although I am not on the Rust team and do not speak for them, I have enough familiarity Rust's funding situation to assure you that there is no budget available for developer evangelism. :P

The_rationalist• 7 years ago

It's not obvious to everyone yet but the modern language that will become the most mainstream is obviously Kotlin. It has almost all modern language features, a ton of justified syntax sugars, and is 100% compatible with the Java ecosystem making it the most production ready among "modern languages"

The only other language with such a property is to my knowledge, typescript.

hnaccy• 7 years ago

Everything about TF on Swift says zombie project.

I'll be amazed if it ever gets any significant foothold in the ML world.

Abishek_Muthian2• 7 years ago

And we havent even touched on the elephant in the Room - Swift may soon hook into OSX Metal

Not sure whether I understood your statement correctly, but you can use Swift for using metal API already. Swift is not the bottleneck for TensorFlow, PyTorch like open-source machine learning library appearing for macOS based on metal, it is the closed nature of mac ecosystem itself.

Look at the flak Google, Facebook get regarding these projects in-spite of them being open-source. If an equivalent metal (only) based open-source machine learning of this scale did pop up with Apple's help; it may not go well with the ML/DL community as sharing & collaboration is valued high here. Apple has even started to publishing open ML papers to attract talent.

Then again, Apple doesn't get as much scrutiny as Google/Facebook does with the tech community, take a look at any HN thread on GoLang/Swift; the discussion on the former goes to the company rather than the programming language as in the case of latter in-spite of both being open-source.

sandGorgon3• 7 years ago

I'm aware that you can hook into metal using swift already. I meant the shader language (which is a variant of c++), though there are issues and proposals to make it swift centric.

ComputerGuru• 7 years ago

I don’t know why you are painting this as Swift vs Rust when they are apples to oranges. Why so defensive? They’re drastically different languages operating at different abstraction levels with different (though overlapping) use cases, with different motivations and design goals.

hellofunk3• 7 years ago

Swift for writing Metal shaders? That's the first I've heard of this, and honestly I find that doubtful but what do I know.

k__2• 7 years ago

Good!

I think Rust will eat more and more of the stack.

It has similar performance as C/C++, but feels more higher level. It's a drop-in replacement for C and is developed by a company that devs like.

madmax96• 7 years ago

Rust is not a drop-in replacement for C, it’s a drop-in replacement for C++.

pnako• 7 years ago

It's a drop-in replacement for neither since it's an entirely different language. C++ went to some great pain to be a drop-in replacement for C; you can "drop in" a C++ compiler in your toolchain and still compile your C code (mostly).

zabzonk• 7 years ago

If it is a "drop-in replacement" the rust compiler should be able to compile C code - it can't.

lewq• 7 years ago

Nice article!

yubozhao• 7 years ago

I think it is also worth examining what type of production load and its importance within the business.

For mission critical production usage, I think to use high performance systems/languages is pretty good starting point. With the assumptions that you update your model conservatively(not often), there are enough engineering resources to maintain and 'port' models from python, since most data scientists are trained with python. I think in this type of workload and context, it makes sense use Rust for deployment.

Another type of workload I think it is much more common. It is the experimental projects that data scientists are trying to discover does those have enough ROI to be part of the production system. Those projects and deployments are requiring a quick turn around on iteration cycle. I am not sure Rust or even Swift are good tools, when typical data scientists are not well versed in those. Not to mention, usually in this setting, they don't have a lot of engineering resources they can use. Python is still the go to option for this type of work.

I think the article has the right intention, speed up ML production. For the experimental work setting, I think we can have the cake and eat it too. Data scientists, still use python and generate production ready deployment service without help from engineers.

We create an open source python lib/platform called BentoML(www.github.com/bentoml/bentoml). BentoML makes it easy to serving and deploying ML models in the cloud, from ML model to production API endpoint with few lines of code. You can try it out at this Google Colab notebook(https://colab.research.google.com/github/bentoml/BentoML/blo...)

BentoML works with multiply ML framworks(Tensorflow/fastai/pytorch/etc) and could generate different distribution formats (docker/AWS Lambda/CLI/Spark UDF) for your serving need. We also support custom runtime backend. Feel free to ping me or ask questions in our slack channel. We are pretty active there.

Taking ML to production with Rust

Related Stories