simag

This is a follow-up of the 2021 post: https://blog.kraken.com/product/engineering/oxidizing-kraken...

We originally introduced Rust (back in 2018) as a small experiment alongside existing systems, mostly to validate safety and performance assumptions under real production load.

Over time, the reduction in memory-related incidents and clearer failure modes led us to expand its use into increasingly critical paths. This post focuses less on “Rust is great” and more on the tradeoffs, mistakes, and organizational changes required to make that transition work in practice.

Happy to answer questions about what did not work, where Rust was a poor fit, or how we handled interop with existing systems.

If by excessive typing you mean complex trait bounds and the like, it's really something isolated to the most generic libraries, and reasonably rare. Most application code is simply typed - take a struct or an enum, return another. So it's not been a problem.

We don't use Copilot or any tool that sends our source code because of our security posture, defense in depth, etc. If and when there will be an offering that lets us run the server component and provides significant productivity improvements, we will subscribe.

With hindsight, considering the cards we were dealt, there's not much I would have done differently. If I had known better before, I would have ensured stronger buy-in because after a while our internal stakeholders were often pushing back on the effort, and that led to concessions where throw away code in the legacy systems was built even for weak business outcomes.

Overall I share your concerns. Having the right reasons to rewrite is key. I believe this blog[1] about software as theory building does a great job at describing the challenge with software gardening, and the times where a rewrite is the solution are few. Even then, it's critical to handle the rewrite in ways that can work - in our case, we chose to progressively eat the legacy software without making major changes when we could avoid them. The legacy software we had was mostly the results of one man heroics and traded off performance and availability for correctness and security. It also was designed to be maintained by a small group. Solid choices if you are early Kraken - but as many successful startups, we were victim of our success and we needed it all.

When it became clear that we had to rewrite the stack (the 2017 3-days shutdown happened just before that realization), those in charge at the time decided to experiment with Rust. It was a crazy bet in early 2018, it was still Rust 2015, no NLL, no async, far rougher ecosystem. The fact that it became successful enough to warrant pursuing a full rewrite is to credit on some lucky hires who made it a success.

In that regard, Rust was a very strong talent magnet. In my experience, having hired 200+ Rust engineers over the last 5 years, there are a few kind of engineers attracted to Rust: (1) some just like shiny things/hype, (2) some are perfectionists and never complete a project, (3) some just are doers who have found that Rust is a particularly effective language.

Overall, Rust has been a great to hire for. Many engineers out there want to use Rust, even if it's their 1st Rust professional experience. We were also known in the Rust community for hiring for full time Rust, probably also the place currently with the highest density of Rust talent (there are massive companies with more Rust devs, but smaller % overall). Budget wise, our Rust engineers are not paid particularly better than other engineers in the company, but the compensations at Kraken are generally in the higher tier.

At the risk of sounding boastful, in my experience Rust is reasonably easy to learn for experienced/strong developers (we have some very young outstanding Rust devs as well, most of the time they learned before joining). Average developers struggle and may never become productive. Again, we have an engineering excellence culture so it is okay for us, YMMV.

Re scope, yes we use Rust for everything in the backend, including CRUD type of work like Web APIs. We've found we're at least as productive than other languages (Go / Java+Spring / Ruby / PHP) while having far fewer incidents, and easier maintenance / cheaper KTLO. Rust's ability for reuse is excellent which means that there are very strong network effects when having more services in Rust, including the Web layer.

A nice "side effect" of a full Rust stack is that our p99.9 latency internally is usually stable around 3-4ms for most operations, even though multiple services are involved. That's coming from a much higher baseline with much more deviation across operations providing the same functionality (60-100ms).

Regarding your own rewrite discussions, you're not going to be convinced by a post on HN, I'll just say that I am very reluctant to even think working at a workplace that doesn't predominantly uses Rust. I've been in the industry for 20+ years, across many stacks and there's a before and an after Rust for me. It has been a super power and made our life easier. It makes it easier to model business problems thanks to algebraic data types and their usage for error handling (versus inheritance), traits allow to abstract behavior better than OOP-style interfaces, the absence of data races is a game changer for multi-threaded code, dependency management is trivial, the ecosystem is rich and things work well. A lot of these are properties found in other languages but no other has the same full package and is on its way to become mainstream.

[1] https://www.baldurbjarnason.com/2022/theory-building/

Completed a full rewrite of many components of the Kraken.com backend in about 4 years.

The new system is around 1.5M loc of Rust. There was no serious alternative to rewriting, sometimes you find yourself in a corner and need to fix issues, and pay the price.

I wrote about it 3 years ago here https://blog.kraken.com/product/engineering/oxidizing-kraken...

Everything in that blog post still rings true and hindsight is that it were were right. But it was a massive grind and required extreme dedication to get it done, for a variety of reasons that work was very taxing.

We also didn't stop feature development and kept the two systems running concurrently (which explains why it took so long, also growing and training a new team 10x the size took time, so there are many factors).

I'm also against rewrites if I can help it, but reality is complex and sometimes we can't help it. Now however, since we removed the last pieces of legacy that were preventing larger DB schema changes (or required massive, unreasonable changes to the legacy systems), we've been shipping faster and easier than ever and caught-up on a lot of the accumulated backlog, including some of the more ambitious projects that were unthinkable in the legacy systems due to limitations.

Kraken | Full remote, globally distributed | Full-time

DESCRIPTION: Kraken is one of the top cryptocurrency exchanges worldwide (and becoming more than that!). We build a lot of our modern backend code in Rust in various teams, and the Frontend in React (and React Native) and Angular. There are many more roles as we're hoping to hire 100-200 engineers (developers, SREs/DevOps, data...) in 2021. Make sure to check our careers page.

Web/Mobile roles: React, NextJs, TailwindUi, Angular

Rust roles (backend or desktop): Core Backend (the core services of the exchange), Bank (the neo-bank we are building), Futures (the Futures/derivatives trading platform), Trading Technology (trading related services). We have a trading desktop app written in Rust with the Iced framework (https://cryptowat.ch/apps/desktop). We hire backend, test and site reliability engineers starting at 2 years of experience. For dev roles, Rust experience is mandatory, but significant pet projects counts. I wrote a blog to provide more context on what the Core Backend team does and the profiles we are hiring: https://blog.kraken.com/post/7964/oxidizing-kraken-improving.... Please read the blog before applying!

LOCATION: We are fully remote+global and have been for years. We also have some offices in around the world for those who like these.

ESTIMATED COMPENSATION: Depends on the role - we base our compensation on the top percentiles of US compensations and apply the same ranges wherever you live. All our offers have option packages. For an idea of the compensation, head to levels.fyi and check Google, Amazon or Apple comps per levels.

REMOTE: Some teams have timezone restrictions: Futures are centered around UTC, other teams are mostly Eastern Europe to US West Coast.

The Core Backend team is planning to expand its presence in APAC and is hiring outstanding engineers who are senior + self-driven + advanced in Rust, who agree to adapt their working hours to maximize overlap with US or EU at least until the group gets big enough to become fully autonomous.

VISA: No visa sponsorship

CONTACT: You can apply on our career page at https://jobs.lever.co/kraken or you can DM me on Twitter (@simach) though I may take a few days to reply.

Kraken Digital Asset Exchange | Software & Site Reliability Engineers | Remote | https://www.kraken.com/en-us/careers

Kraken was founded in 2011, being a pioneer exchange in the crypto market. Kraken is in rapid expansion of its engineering teams. At Kraken we have a remote global workforce, allowing our employees to have more freedom in choosing how to organize their routine and balance work and personal life.

March Highlight: (Senior) Backend Engineer - Rust

Check out our blogpost for further insights into Rust at Kraken: https://blog.kraken.com/post/7964/oxidizing-kraken/ - I'm the author of that post so feel free to reach directly for any question.

We're also hiring software engineers with the following skillsets: Backend (Rust/Go/Typescript/PHP), Frontend (React/Flux/Typescript/Sass/Less/NextJS) and many more.

Please apply directly on our careers page at https://www.kraken.com/en-us/careers.

We're using thiserror, and haven't explored macros that generate an enum from the Result type. I remember there are crates exploring this, but this is something that would really benefit from compiler support. Ocaml's polymorphic variants appear to provide an interesting solution here.

Author of the post here - feel free to shoot any questions.

Also see thread on r/rust: https://www.reddit.com/r/rust/comments/lnkx9h/oxidizing_krak...

I disagree with Stallman's skepticism (I do think it harms children and I don't think children can make that choice). But that one liner does not make him a pedophilia advocate. That makes him someone who is wrong on that topic.

It's a 3 person side-project over 3 months. I suppose he means that Rust is to thank for this level of polish and reuse of 3rd party libraries. My experience with Rust is similar and there are many projects I would consider impossible (practically not technically) without Rust.

Programs like shells or servers are expected to perform well and usually cannot afford the GC tax (more memory usage and pauses). I wouldn't use a shell written in a GCed language. That would leave them with non-GC'ed languages. C or C++ simply don't have a usable equivalent of Cargo/crates.io which makes 3rd party reuse slower, and then they have much weaker execution safety guarantees.

Kudos!

When Rust 1.0 was announced I had a look. I was very surprised because that language was very different from my previous look, when Rust still had GC and green threads (and a syntax with mystic sigils).

Rust 1.0 looked like it could be a good language, but coming from mainstream languages with a richer ecosystem -- such as Java/Scala or C# --, Rust lacked many common libs. For instance, a high-level approach to non-blocking IO, which is something I need when writing servers. There were other issues -- lexical lifetimes often meant one needed to jump through hoops to please the borrow checker. I thought Rust would need a lot more time to mature enough to be a serious contender in my PL list. The next year, the Zero-Cost Futures post[1] by Aaron Turon showed up in my news feed. It seemed interesting, but it was only a plan. A few months later Tokio was born and, again, it seemed interesting. But there's no way it was ready, was there? I went back to business as usual. I kept hearing about Rust, but I had no intention to use it in the immediate future.

Last year, I was offered a full-time Rust job out of the blue (because of my experience in building distributed systems; I had to learn Rust as a part of the job). I was interested because, in general, I was looking for a change -- but I also wanted to make sure I wouldn't spend energy on a language that wouldn't be relevant in 10 years (at least, according to my intuition). I've had job offers in other emerging languages (or old but trendy FP languages) in the past but I don't think they will becoming mainstream, and I want a language where 3rd party libraries for every (modern) protocol, API, database, etc, are complete and safe (and that's more likely to happen with mainstream languages).

I evaluated Rust again to see how mature it was. I was very surprised at the speed of development of the language and of its ecosystem, especially compared to other PL communities I had been part of. The energy in Rust reminds me of Java's better days (but with a modern language). There are librairies for everything, the language and compiler is surprising stable and mature (YMMV). I believe this is a testament to Rust's productivity and reuse. Rust is not perfect, but it's trying to reach an interesting trade-off. It is already very usable considering how recent it is (obviously taking advantage of LLVM).

In my experience, projects with such hype are often just good at marketing. You try them, and they have a lot of issues that they (sometimes) fix later; I'd say it was the case for Cassandra, Docker, Kubernetes, TypeScript (and more). With Rust, I started almost immediately on edition 2018 and found the tooling, ecosystem, language and compiler to live up to its hype. Sure, compile times could be faster, the RLS/IDE support is still a work-in-progress (though usable), but I'd happily trade compile times for fewer bugs at runtime, and most of all, the trajectory looks great; coming soon: async/await, const generics, GATs, and better IDE.

So -- congratulations to everyone who had a part in making Rust happen, and happy birthday!

[1] https://aturon.github.io/blog/2016/08/11/futures/

Interesting post that show some current limitations of Rust, especially when not being able to control the design entirely.

Rust's ownership system makes it harder to deal with the kind of dynamics wlroots exposes... It seems to me that the issue here is that there is no consensus (e.g library) on how Rust should deal with those issues, so if you want to just to provide bindings to a dynamic system you end up writing middle-ware that deals with that kind of dynamics in a Rust idiomatic fashion (taking advantage of ownership) instead of focusing on the task at hand. It somehow destroys Rust's promise of productivity.

A Rust project of mine[1], in an early stage and currently paused, deals with those issues (probably in a way similar to the Handles described in the post) by providing Service smart pointers (i.e, Svc<T>) and a component-framework that helps deal with the dynamics of a service being unregistered (e.g, a monitor unplugged, or a dynamic library unloaded) ; that component framework helps you be sure that if your struct contains a Svc<Foo>, that service is still available when you're called, or the component reaches an invalid state.

In general, it seems to be an area where Rust's ecosystem is still very early and would benefit from more input (such as this post) and consensus.

Regarding the callback-hell issue, that "dehandle" macro looks very much like async/await, and it looks like it could be implemented either in terms of async/await (still unstable) or generators (even more unstable).

Hopefully similar projects will be more likely to succeed as Rust and its ecosystem mature.

[1] https://github.com/magnet/socrates-rs

Thank you for your kind words, and kudos to you for finding that buried submission ;-)