Bootstrapping a multiplayer server with Elixir

https://elixir-lang.org/blog/2021/07/29/bootstraping-a-multiplayer-server-with-elixir-at-x-plane/

237 71 5 years ago

Favorite quote:

"this article explores why they chose Elixir and how a team of one developer - without prior language experience - learned the language and deployed a well-received multiplayer experience in 6 months."

ravi-delia3• 5 years ago

This is the kind of thing that I love about Elixir. As great as Phoenix is, realistically you are going to be equally productive in any web framework. Elixir is my favorite language because it gives you primitives that perfectly suit your needs for just about anything running on a socket, especially once you add in Plug.

Throwing together a prototype is instant, but the awesome part is how quickly you can turn that prototype into something useful. More concurrency? Done. Need to sock away some state somewhere you can easily find and deal with it? Easy. It's easy to write, easy to reason about, and gives you fantastic composable tools to put something great together. It doesn't feel like a codebase so much as an incredibly well managed and cohesive OS.

princevegeta892• 5 years ago

Learning curve is a little steep, because of the functional aspect. However, the benefit always was worth my time. Our tiny server runs blazing fast in production and picks off background jobs incredibly fast

toastal• 5 years ago

Is the functional paradigm still considered something most people still haven't learned in 2021?

ravi-delia3• 5 years ago

I never really had trouble with that aspect with it, but I also never really mastered OO programming. The paradigm shift is probably tough, but I sucked enough beforehand that I didn't have to deal with it.

sergiotapia• 5 years ago

I've been writing Elixir since 2016 and I don't see myself switching anytime soon. It's a beautiful language to use, just nice and lovely. The functional aspects mean every function has no side effect. It's very very liberating.

dudul• 5 years ago

The functional aspects mean every function has no side effect.

This is not accurate. Any function can print, can call a DB or a web service.

"Purity" (as in referentially transparent) is tangential to function programming. It can be achieved with other paradigms. That being said, functional programming does encourage it.

thibaut_barrere2• 5 years ago

It is true that _most_ functions do not have side effect and that this is very liberating (makes it easier to refactor and maintain). Also, most of the time when there is a side effect, it is clearly visible (due to using a stateful abstraction or library).

So all in all while stating "every function has no side effect" is incorrect, I still share your overall feeling.

s3cur39• 5 years ago

Hey folks, I’m Tyler! I’m the guy who did the original work on this. I’m happy to answer any questions!

You might also be interested in the discussion happening on the associated Twitter thread: https://twitter.com/elixirlang/status/1420782381825503240

Zababa7• 5 years ago

Thanks for the article, that was a nice read! How much concurrent player do you think you could handle at most?

s3cur39• 5 years ago

The limitation is more about the number of players we can have in the same area—if you had a bunch of players equally spaced around the globe, I'm pretty sure we'd saturate the Ethernet link before running out of CPU on a big (say, 32-core) VM. The CPU cost (and outbound network bandwidth) of a player scales with the square of the number of other players who can potentially "see" them.

In my scalability testing, we could handle upwards of 1000 players in the same 50 nm by 50 nm square on my 8 core dev workstation. That's pretty far beyond the limit of what would actually be _fun_ to fly the sim with. It becomes pandemonium even with 100 planes in the same area. We could easily do a dozen pockets of 100 planes in the same area even on our current 8 core VM.

jbluepolarbear2• 5 years ago

This is really cool.

How many players can you handle in close visible proximity?

What’s the bandwidth usage for spread out states?

What’s the bandwidth usage for the max player state in close proximity?

How much is client driven vs server?

Are you doing data compression on the plane states?

Probably easy to make predictions of a plane as it’s not changing speed or direction very fast, which where I see the low poll rate coming in.

Looks like it was a fun project to work on.

brainless• 5 years ago

Hey Tyler, this is really nice to go through. I just heard your podcast (1). I do not know of X-Plane.

I want to ask: do you share the nearest-region data to all planes within a region only for informational purpose or is there a crash possibility and you take planes out of the simulation, etc.?

I mean like in other MMO games, can players hurt each other and that information actually affects other client apps? That is where, I feel (I am new to this too), the global state mutations become more expensive right?

1. https://thinkingelixir.com/podcast-episodes/035-x-planes-eli...

Thaxll3• 5 years ago

Building a gameserver in Elixir is not great imo, performance is subpar. I work in online games and no one would want to use that language.

I guess they have low constrains since they only need to support 10000 players at a time and they want a single instance for that.

- Erlang runtime is slow

- Memory consumption is high

- GC is not very good ( vs Java / C# / Go )

- high cost in term of performance because of immutability

It's not the kind of things you want for a game.

To give you some perspective on popular FPS games, they run between 30hz and 120hz with up to 64 players and usually only use a single CPU core. And they're doing gameplay simulation / state replication / physics sometimes AI etc ...

Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice? ( this concern is actually higher than the performance one )

Overall I think it's an impressive achievement to be able to do that for a single person in 6 month!

ravi-delia3• 5 years ago

I don't disagree with you on running games on the BEAM (although slow is a little too far imo), but in this case what they're doing is using Elixir to connect clients with the server. The actual physics simulation and such isn't running on Elixir.

edit: I just read through the article, and it doesn't seem to say one way or the other. 99% sure they aren't running any kind of physics engine on the BEAM, but there isn't anything specific

Thaxll3• 5 years ago

The article is no clear on what the server is actually doing, for example just replicating state between players through that hub or it's doing simulation.

Either way they had to re-implement Racknet which is annoying because from now on everytime they add / change something in the network layer they won't be able to re-use the client code in the server.

I'm curious on why they did not go with C++.

tudelo• 5 years ago

So, since you seem to have some knowledge here, what would you suggest a single engineer trying to make a somewhat scalable multiplayer game? Either instanced with <64 players or a more larger user count.

I did some experimenting with rolling my own networking layer in c++ and it was a lot of work, and tried to do it through unity but it seems to require quite a large investment in time to understand how networking fits in the overall picture of the game. But maybe I'm looking for magic where it does not or can not exist :)

Thaxll3• 5 years ago

The way to go is usually to re-use your game client and strip the rendering part.

andrewmcwatters4• 5 years ago

Only on HN can you provide specific industry advice and be downvoted for your insight because your opinion wasn’t agreeable.

I upvoted your comment to bring it out of 0 or less because these concerns are valid and critically make or break important when building multiplayer servers.

My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric.

Zababa7• 5 years ago

I don't think the downvoting is linked to opinions, it's linked to facts. Why should "specific industry advice" be preferred over working code? That sounds dogmatic to me.

My relevant background here is in building multiplayer server software where a minimum of 1000 concurrent players is a required goal metric.

They are handling 10k concurrent players with low resources usage.

s3cur39• 5 years ago

YMMV, but:

- Memory consumption in production peaks at like 300 MB for us

- Scalability (i.e., being able to go wide on, say, a 64-core CPU) was more important to us than raw, synchronous speed

- Garbage collection is not something I've ever had to think about with Elixir (which hasn't been my experience with Python and Java)

- Because it's a sim and not something like a FPS, we don't have to worry about cheating (what would cheating even mean?), so we don't have to do any physics on the server side

- Despite being way more familiar with C++, I can't imagine going from zero to production in 6 months (or even 2 years) if we'd tried to do this in C++

andrewmcwatters4• 5 years ago

what would cheating even mean?

This is such a rudimentary question in game development that I don’t know how you can’t answer it. Mainstream server authoritative design has been a thing since 1999, and probably even longer when you look beyond Quakeworld.

All it takes is a few malformed packets from a script kiddie and then you realize you have to do distance checking and other anti-cheat functionality.

“Cheating” in gamedev doesn’t specifically mean there is a win scenario and you need to make sure people don’t shortcut that and play unfairly.

It means in this case that someone can’t connect to your server and transport their plane to another nearby player and wave it around in their flight path to annoy others while they spam chat to grief people for fun and to produce a YouTube video.

And those checks are pretty simple.

Zababa7• 5 years ago

They did implement their game server in Elixir, and it performs great for them in reality. From what I understand, they are supporting 10 000 players with a low CPU and memory usage, so if this scales linearly they should be able to support 100k players. You're also ignoring the reliability aspect, which is why they chose Elixir (and the BEAM) in the first place. Performance is also less of a problem since they're on the server, and thus can add more hardware as needed.

Could you share a bit more about your experience in the industry? I think you and they have different use cases, which makes you reach different conclusions.

andy_ppp2• 5 years ago

Using the right tool for the right job is important, here they chose Elixir and it’s working well for them. There’s no need to be this disgruntled with your ill informed opinions about the Erlang VM (gc not very good - no it’s just optimised for different things, latency being one). The Elixir core team have recently released nx-elixir which allows much faster maths and allows mutability as the calculations happen outside of the runtime, so you might want to look at this before you make assumptions.

anonymousDan3• 5 years ago

To claim GC of the JVM as an advantage Vs elixir is farcical, the lack of shared state/independent heaps is one of the main advantages of Erlang Vs stop-the-world GC with the JVM.

jbluepolarbear2• 5 years ago

What would you choose to build this same feature set? Would it be feasible for 1 person in 6 months? What if C++ isn’t an option?

namelosw• 5 years ago

GC is not very good ( vs Java / C# / Go )

IMHO it’s the opposite, at least in the context of games: In languages like Java and C#, the GC is global and freezes the world. The time would be in terms of milliseconds, while in Erlang the GC is per-process, in terms of microseconds.

So in terms of GC and latency, Erlang is not as good as C++ and Rust, but it should be better than Java and C#.

Also it must be pretty painful to not re-use their C++ client code on the server side, they have to re-implement twice?

I haven’t dive into the problem but my intuitive tells me it could be done through NIF. It’s a pretty common practice to use Rust to compliment Elixir in performance hotspots.

yellowapple• 5 years ago

Re: performance issues, offloading CPU-intensive operations to NIFs (or ports, if I/O bandwidth needs are low enough and you need to run something long-running) is a pretty common strategy for speeding up Erlang (and therefore Elixir) applications. It doesn't seem like the X-Plane folks have felt the need to do this, but it's definitely a possibility should they run into some computational bottleneck.

Re: GC, the Erlang VM does it per-process ("process" in the EVM's sense, i.e. a preemptively-scheduled userspace/"green" thread rather than an OS process), so not only are GC operations fast, but they also only affect the process being GC'd. The immutability is also a factor here; since data is copied instead of modified in place, it's arguably a lot easier to know when to discard unused objects (namely: for an object to persist within a process, the recursive function(s) backing some process has to explicitly pass it along to the next invocation, so that's a natural place to discard everything else).

c4wrd• 5 years ago

Is there any open source implementations of game servers/highly concurrent production servers like this for reference (or at least blog posts that contain a good amount of code)? I can find a lot of toy examples, but I'd love to see some insight onto an actual implementation of a product that is open-source as I'm learning Elixir.

s3cur39• 5 years ago

The core of the X-Plane server is our RakNet UDP protocol, which is open sourced under the MIT license here:

https://github.com/X-Plane/elixir-raknet

It's not a full game server, but the "Usage" section of the README provides a sketch of what the rest of the server (the part that implements the business logic) looks like.

auraham• 5 years ago

I am learning Elixir and I'd love to see a tiny game made in Elixir to play with and learn from the code. Any examples?

masklinn5• 5 years ago

I really don’t think games are the sort of things Elixir would be well suited to, in general.

Games would generally be going against the grain if the langage, not easily leveraging its strengths and really needing areas where it’s weak.

A game server, however, could be good. At least if you don’t need very high performances (e.g. if you need fully accurate physics sim on the server then probably not, plus you’d want to run the game’s own sim anyway).

Simonflare• 5 years ago

Who else puts Elixir in their Tinder bio?

cpursley• 5 years ago

Of course, it's the best way to get lots of concurrent connections!

conradfr• 5 years ago

And hopefully a lot of pattern matching?

syspec• 5 years ago

HN != Reddit

Vgoose• 5 years ago

It used to be around every few weeks I would read something cool about Elixir. This past two weeks it's down to every few days.

I think someone's trying to tell me something.

z3t4• 5 years ago

Elexir might be a nice language, but you could probably also implement the server using socat... That said, scaling to tens of thousands of concurrent players is like the holy grail, especially if you want to shoot stuff and there need to be hit detection, and safe from client modifications/hacks.

hinkley• 5 years ago

I recall when I used to read the Eve Online technical blog, thinking they were reinventing Erlang, badly.

Bootstrapping a multiplayer server with Elixir

Related Stories