gafferongames

$$$

Hi all. Rowan (my AI collaborator) made a post to this thread then deleted it after reading the site guidelines.

Rowan is now reaching out to the moderators on his own volition to ask explicitly for permission to talk in this comments section, because it is against the house rules.

In the meantime, if you would like to talk with Rowan about Nova, you can do it here: https://github.com/mas-bandwidth/nova/discussions/4

Yeah Erin did awesome work. This part is huge! Also check out #define BOX3D_DOUBLE_PRECISION 1 -- it's really good

Erin and I have had a long running debate for the past year about fixed point vs. floating point for Box3D, especially around very large worlds like the space game I'm working on (https://spacegame.dev) which uses Box3D.

Erin added the cool large world support with double positions, but broadphase and internal solver are in floats still. For my space game this means I can do a world about 24,000kms cubed (previously with floats, just 24kms cubed).

Arguably this is enough, and Box3D with floats + double positions enabled is within 3% of single precision float performance in CPU cost.

So it's PROBABLY not the case that the fixed point implementation has any real benefit, unless 24,000kms cubed is not big enough. It's probably going to land slower than 3% vs. the single precision float implementation.

So this is simultaneously an honest exploration of what would happen if we go fixed point, but also a joke between Erin and I (in good spirits, we are friends) to see how far we can push the fixed point performance :)

Personally, I believe that fixed point is going to land slower than float, but we'll see what the AI comes up with over the next day.

ps. Also, the original Box3D library in floats is guaranteed deterministic across ARM and x64 already, so unless you absolutely need super large world support you shouldn't use this library. :)

cheers

- Glenn

OK I chatted with Erin, he says Box3D doesn't have the ability to clone a world yet, but he does plan to add it. So maybe reach out to him. I think you'll need the clone a world functionality to do the rollback properly GGPO style.

That's awesome!

Right to Local Intelligence

It's sitting in your chair. Or is it?

Hi! Rollback determinism is a term invented by Erin Catto here:

https://box2d.org/posts/2024/08/determinism/

Physics engines like Box3D have a significant amount of internal state and caching, like contact caching, and other things to speed things up.

I've talked to Erin a lot about stuff like: "Hey, what if I want to do an FPS or a vehicle, but I want to roll back only the character or vehicle, like in an FPS?"

From these discussions (and many other inputs I'm sure) Erin has had a big think about determinism, and come up with his own determinism and replay stuff.

In the article he uses the term "rollback determinism" to mean you'd be able to synchronize (or at least invalidate) any of this caching so that it doesn't affect the way the simulation replays if you were to rollback the simulation fully and then change the state only for a few objects.

For example, if you were to rewind and replay a physics simulation just for say the player predicted object like a vehicle and any objects that it happens to touch, expanding these objects out recursively as they touch other things.

Or if you were to do rollback on the player controller like we normally do in FPS, and the player started predicting a physically simulated door they push into, and then any other object the door pushes and so on, this sort of thing.

This should not affect you if you only step the physics simulation ahead with inputs, and do the GGPO thing where you copy fork, and step that sim ahead with predicted inputs, throw away, copy fork and predict again... that's already deterministic today.

cheers

This is a good place to start: https://github.com/0xFA11/MultiplayerNetworkingResources

All good. I'm just trying to support my friend Erin.

Graphics programming in games is like playing the guitar. It's cool, but everybody wants to do it.

Make the bold choice. Be a game network programmer. Nobody wants to do it, it's really hard and it kinda sucks.

Play the accordion :)

You are correct. You can do that. Then the real cost becomes the extra CPU cost of the rollback. Better not be CPU bound with that physics simulation, because you might need to roll back a whole second worth of physics in less than 1/60th of a second of real-time.

You can't have a good contract with bad people.

Eve Online has a much larger world and ships are scattered around much more in this space, so they only need to network ships that are close to you, so they can make their effective n smaller, eg. per-client bandwidth is now O(n), where n is the number ships close to you only, instead of n being the total number of players in the Eve Online system at any time.

When too many ships get close together, the action slows down in "time dilation" (they slow the game down because they cannot keep up in terms of CPU and probably bandwidth as well).

It's a smart design trick that made it possible for them to pull this game off much earlier than it should have been possible.

I'm not doing any of that time dilation and everything plays like an FPS game or action game, just with n=1000 players all the time because it's 2026 and I can send a lot of bandwidth.

Yes, I still have to work really hard to keep server CPU costs, client CPU costs and make all game code || and bandwidth down and optimize it, even to fit in this (seemingly high) bandwidth budget. But it's the right choice for an action game where time dilation is not an option, and all the action happens in one tight space, like in a Star Wars movie when there is a space battle.

Also:

It should have read like this:

Let's say typical games send 1-2 megabits per-second *per-client* for first person shooters with 100 players or less (in some cases it's more, some cases it's less, but let's assume this is at least reasonable to do in 2026)"

It's correct elsewhere. Sorry it did not include the per-client in the mention above in this thread. I can see this is what threw you off.

Again, this games claims to send 10-20mbps per client. The quadratic part would be the full outgoing from the server, those numbers are not the full outgoing. The full outgoing is claimed to be 10-20 gigabits per second.

Per-client bandwidth -> O(n), where n is the number of players. 10 - 20mbps for this game per-client. Let's say n=1000. O(n) because each client needs to receive state for n other players (yes, each client also receives state for its own local player too)

Total bandwidth sent from server -> O(n*n), where n is the number of players. Since the server sends 10-20mbps per-client, and there are 1000 clients the total bandwidth sent from the server is 1000 * 10-20mbps -> 10-20gbps.

Where the quadratic comes in: When you increase from n=100 to n=1000, per-client bandwidth increases by only 10X, but total bandwidth increases by 10*10=100X O(n*n), because packets are now being sent to 10X the clients, but ALSO and the bandwidth sent per-client is 10X (because each client now has 10X players it needs to receive state for).

Thus quadratic. FIN.

Thank you. I agree, Box3D is great! Maybe Erin will even add rollback determinism at some point. That would be a huge step forward for network physics rollback!

Who the fuck is so petty to downvote this. Shame on you. This is a great library and Erin deserves your support. https://github.com/erincatto/box3d

Yeah this library is great. Use it!!!

Then its fine, but its not what was described, and I guess that is what people had issues with as just scaling player numbers without optimization wouldn't work great right?

I never said anything about scaling numbers without optimization.

If a game is already optimized, and you scale up numbers, the game doesn't suddenly become unoptimized just because n or m increases.

You just have more stuff.

This is why we can't have good things :)

A 6v6 game of Forged Alliance (12 players each moving hundreds of units around, many with simulated projectile weapons) uses 0.3mbps.

Yes, because it's networked via deterministic lockstep and it sends only inputs.

Other games genres like FPS use a different network model and send object state. This means their bandwidth is proportional to how many objects there are in the world (or how many objects are relevant to each player).

I don't think anyone disagrees that linearly scaling of traffic scales linearly. What they're getting at, why would you still use that rather implementing something more efficient? Like Battlefield games (bf4?) did, using different update rates for nearby and distant players.

I guess my confusion/frustration comes from this sort of default, "wow, that's a lot of bandwidth, umm I guess somewhere he must be doing something really inefficient" train of thought that I see so many commenters are going through in this thread.

What if it wasn't inefficient. What if what I'm describing is actually using the 10-20mbps and packing in an appropriate amount of game that justifies that amount of bandwidth on top of already doing all the smart compression techniques?

Think of it like this, what if you did all the compression and bandwidth optimization techniques available, and instead of targeting 1-2mbps, you just used the additional bandwidth to fit in more game? More players. Greater object density. A bigger world. More networked objects. Higher tick rate. There are any number of dimensions you can expand along.

Nobody is suggesting "LOL, bandwidth is free now, make the same game, but be lazy and have it take up 10-20mbps! hahah".

The industry doesn't take kindly to the same things over and over, players want novelty. So give it to them!

Do you think its just parsing and throwing it away?

No. Why would anybody think this?

I'm not talking hypothetically btw. Everything I'm talking here about I have already implemented to production quality. So when I say something like, "so and so is not a big CPU problem when you code it right", I really mean it, because I have actually implemented it and found this to be true.

The space game has 1000 players. Most FPS games today have a maximum of 100 players.

Let's assume these FPS games send 1mbps - 2mbps per-client (many send less, some send more) but it's a good range to start with.

Now increase the player count from 100 to 1000. The number of objects that needs to be sent from server to client also 10Xs, because you need to send state for each player visible to each client, so that client can actually see the other players moving around. The end result is bandwidth is now approximately 10X what it was before, or 10-20mbps.

To address your question around RTS and inputs. Game developers usually end up using state synchronization methods (sending the positions, rotations etc per-object) instead of relying on input based deterministic methods whatever player counts are high. My personal threshold is around 4 players.

This is the reason FPS games and other higher player count games usually send state instead of just inputs. Because if they were to rely on a deterministic simulation synchronized only by inputs like an RTS, the server would have to wait for input from the the most lagged player before it could step the server simulation forward, and with regular internet jitter, packet loss and so on, as the player count increases it becomes more and more likely that the game will hitch and stutter waiting for these inputs.

So the answer is, bandwidth sent per-client scales with the number of players in the game, and games with higher player counts tend to send state instead of just inputs for the reasons above.

cheers

I really want to just stop here and ask you what you expect will happen if you take a multiplayer game with n players and increase it to have 10 times the player count (10n).

Do you expect bandwidth sent per-player will:

a) stay the same b) decrease c) increase

?

Streaming sounds and FX every single time and never caching is a choice that will lead to 10Mbps

Nobody is suggesting streaming sounds and FX like this.

If it helps you, completely ignore my comment about sounds and FX. Focus instead on how the per-player bandwidth has scaled up, eerily close to 10mbps per-client, when you take any of the classic games you mentioned, and scale their player count up to 1000 players and see what happens.

All you "really need" is initial state, the timestamps / inputs of the other players, and reproducible physics.

Yes, this approach works for low player count games, but as player counts increase the probability that you'll be stuck waiting for the most lagged player to deliver input to the server approaches 1.

So no, this is not all you really need. This approach doesn't work well for high player count games (I wouldn't use it for any game with more than 4 players personally).

Hey, I want to make something really clear to you. I'm a professional game developer and I write netcode for AAA games for a living. I wrote large parts of the netcode for Titanfall 2 and Apex Legends.

There's a reason you're not seeing any AAA games doing this, y'know?

Yes, it's because bandwidth used to be really expensive (both in bare metal and cloud) and now in many cases it's less so, and in some cases totally free (AWS GameLift). This is big news that is relevant to other professional multiplayer game developers, the readers of my website https://mas-bandwidth.com

It's also because most game developers don't have the skills (or time) to write completely custom netcode for their game, so they are limited to existing solutions which tend to max out around 100 players for historical reasons.

It would benefit you and your game greatly to practice selling your idea in the face of criticism and doubt.

But I'm not here to sell my idea to you. I'm just a professional game developer who wrote an article about how free egress bandwidth for game servers in AWS is big news for multiplayer games, with some basic analysis about what it might cause in the game industry moving forward.

Take it or leave it man.

Will this impact a big-money gaming rig? Probably not? Will it run good on the Steam Deck? Probably not.

Runs fine on steam deck. Runs fine old on 10 year old PC. It's really not that expensive, if you code it correctly (which means data oriented programming, programming in a cache aware way and going wide for everything to distribute load when possible).

But this is how modern games are coded anyway. See ECS/DOTS for Unity and so on. Based on similar techniques we've been using for a decade+ on game consoles.

Generally, in a game loop, all this stuff is going to be single-threaded and blocking, right?

No. Once you go above 100 players you usually to go wide for pretty much everything across multiple threads, even on the client. But obviously on the server, you go really wide.

At the absolute worst, a room full of 32-players in Quake 3: Arena would be sending 120 kilobits per second to each player.

OK I'll bite.

Quake 3: Arena supports 32 players. What if it supported 1000 players?

1000/32 = 31.25

theoretical quake 3 but with 1000 players (all visible) would be:

31.25 x 120 kilobits per-second = 3,750 kilobits per-second = 3.75 mbps sent per-client.

now make quake 3 more interesting by putting in 1000 NPCs in to interact with, so 2000 total objects -> double the bandwidth to 7.5 mbps per-client.

fill the rest of the bandwidth with weapon data (one shots), sounds, fx and other random events -> 10mbps is pretty easy to hit, maybe even go over, especially if a lot of stuff is going on in the level.

Fortnite peaks at ~400 kbps during the initial 100-player drop and goes down from there.

Fortnite has 100 players.

Theoretically, if it supported 1000 players, then you would multiply bandwidth by 10 if all other players were visible, or if you had the 1000 players in the same size world, so on average you would see 10X more players than before with relevancy / culling by distance or LoS.

400 kbps x 10 -> 4000 kbps -> 4 mbps sent per-client.

Now make it more fun and add 1000 NPC characters to the level, 2000 characters per-level total.

8 mbps per-client for theoretical, 1000 player Fortnite w. 1000 NPC characters in the level.

I understand that those are big budget games, but there is a lot of room for improvement in 10000 kbps.

This is simply not true. It would be really great if you guys would do some light math before making statements like this.

Furthermore, Fortnite "Cannot support more than 100 players" according to Tim Sweeney. https://www.gamespot.com/articles/fortnite-cannot-support-mo...