Automated testing in Bevy

https://chadnauseam.com/coding/gamedev/automated-testing-in-bevy/

by ChadNauseam • 4 years ago

107 81 4 years ago

Jensson3• 4 years ago

Bevy uses the ECS pattern to organize game code, and it occurred to me that ECS substantially alleviates the spaghetti-code problem that makes testing in Unity and Unreal so terrible.

Both Unreal and Unity uses ECS.

guzik2• 4 years ago

Seems like the author never truly worked with these engines? Also, I never had problem with unit testing on Unity in pure C#. Even when it comes to complicated scenario there is always a way to separate your GameObjects from game logic.

harles3• 4 years ago

I think this comment and its parent have misunderstood what ECS is.

ChadNauseam4• 4 years ago

I have spent way too much time in Unity. (I was a Unity developer for pay for several years.) I suppose I didn't do a good job explaining the differences between what Unity does and an ECS approach in my post, but if you'd like to get a feel for what an ECS you could try out DOTS. (Although DOTS is kindn of unpleasant to use compared to Bevy.)

tarsiel• 4 years ago

Unity at least isn’t a true ECS architecture, more just Entity-Component. You can write game logic in an entity with Unity, whereas in ECS that would have to live in a System

runevault• 4 years ago

Unity is working on a full ECS system as part of a larger DOTS initiative, but it is still pre-alpha last I knew.

metadat• 4 years ago

What is ECS? Assuming it's not AWS Elastic Container Service :>

worble• 4 years ago

Entity component system

keyle2• 4 years ago

Around 2007 I believe the game industry came up with a semi standard way of building games that somehow alleviated a lot of issues in producing complex games... aka spaghetti loops of hell.

There are a few ECS approach and to this day it's still vague who does it best (depends on your goals).

One I found interesting is the "sparse entity component system".

keyle2• 4 years ago

Thanks, I rolled my eyes and had an issue with the author's statement.

My guess is what he meant was that the Bevy approach is cleaner to test upon.

harles3• 4 years ago

Can you point me to what part of Unreal uses an ECS approach? I’ve never seen this. I know Unity DOTS added ECS features, but that still lives in a hybrid world.

Jasper_3• 4 years ago

This is where the unfortunately-chosen name "Entity-Component-Systems" (ECS) comes back to bite us. Unity's GameObjects are Entity-Component (EC): you have an Entity (GameObject), onto which you attach Components (MonoBehavior). Unreal is similar: you have an Entity (UActor), onto which you attach Components (UActorComponent), though you can also have plenty of code living outside the UActorComponent system in Unreal. Entities are objects, and Components are code/data.

Entity-Compontity-System (ECS) uses these nouns differently; an Entity ends up being little more than an identifier for a large number of Components, which are data structures, and Systems are the code that modify groups of Components. "System" here is a noun, not a descriptor of the practice. The usual analogy is similar to a relational database (in fact, the point of the system is to achieve wide-scale performance like RDSes do): Entities are rows, identified by a primary key, Components are columns, and Systems are the SELECT/INSERT/UPDATE/DELETE queries which acts on the data.

Unity has a new-ish ECS framework under the DOTS initiative; Unreal has no parallel. Very few games are using these.

Jensson3• 4 years ago

In unity the methods "Update", "Start" and so on are the system part of ECS. That you write their implementations alongside the data doesn't really matter.

namrog84• 4 years ago

Unreal 5 added true ECS called MASS and its really quite great.

It uses different terminology such as fragments. Traits and others but its incredibly powerful and scalable.

This is a separate system then the ec of actors and components.

stuckinhell• 4 years ago

I've been looking into Unity and game development in my spare time. It seems writing tests for game code is incredibly dependent on the style used to develop the game.

If you are doing everything by hand in C#, its super easy. The moment you introduce editor workflows or third party libraries, I have no idea how anybody can write sensible tests.

Using Unity is like a weird mix of no code tools and code.

guzik2• 4 years ago

Using Unity is like a weird mix of no code tools and code.

Haha, great summary of my 5 years experience in Unity. Whenever I am too lazy I am switching to ECS/MonoBehaviour, then I regret of not having properly testable architecture.

Jaxkr• 4 years ago

ECS is a competing system to supersede MonoBehavior

andrewallbright• 4 years ago

Writing automated tests for video games is some of the most fun programming I do.

Goz3rr• 4 years ago

The author mentions Sea of Thieves and Mindustry, but I think Factorio is worth a mention too (seemingly as usual when something game related comes up on HN). They do TDD and talk about their automatic tests in several blogs:

https://factorio.com/blog/post/fff-366 https://factorio.com/blog/post/fff-288 https://factorio.com/blog/post/fff-186 https://factorio.com/blog/post/fff-62

Jasper_3• 4 years ago

Riot Games also has an automated testing framework: https://technology.riotgames.com/news/automated-testing-leag...

I still think the best combo is a mix of automated and QA. Software in general went through a large decline in quality after QA was laid off across the industry, and in my experience, automated testing just tends to find different kinds of issues. Also, playtesting is also not the same thing as QA, but I digress :)

tmpz22• 4 years ago

What a lot of people don’t realize about QA is that it’s also a point of Empathy to your end users. Some might think it’s the PMs job to think as an end user but in my experience PMs are so bogged down with other sales adjacent duties that they rarely if ever can provide that pillar.

This is a core issue with for example Google products where MANUAL PROCESS BAD seems to be a religious bastion.

bongobingo1• 4 years ago

Wonder what their coverage is in this - or if they even still use it. There are many occurrences of "how does this ship" bugs in League where damage numbers don't add up or status's have no actual effect, etc - sometimes hanging around for many patches. Not that I don't understand how hard it is to test the big ball of mud that (10+ years old!) game engines end up being.

treeman79• 4 years ago

Recently Inherited a code base with almost 20000 tests. Many very detailed. Most of them were so mocked out that at end of day it was just ensuring the mock library was well tested.

towawy• 4 years ago

I strongly agree with this. You need someone around that actually uses what you develop, be it dogfooding or structured QA.

I wonder if there is any books on game/software QA akin to Masters of Doom or Stay Awhile and Listen? Those pretty much skip over QA, it's usually just the devs playing their own game (which anecdotally is usually biased and of limited use).

keraf• 4 years ago

The industry is changing towards more automated testing, I heard of a lot of AAA studios implementing/creating testing frameworks. It was very uncommon back in the days, they just threw a bunch of badly paid and overworked QAs at new builds. Games are very complex, covering a wide range of behaviours in environments with a lot of randomness is difficult. With automation, the predictable behaviours (for example, testing that climbing a ladder works) can be offloaded from QA and they can focus on finding more random issues with playthroughs. I guess there was also a push from engineers without a gaming background (and being more used to TDD) that have been recruited in the industry throughout the years.

senko• 4 years ago

Here's also a good overview how Croteam (creators of Serious Sam and The Talos Principle) approach automated testing: https://m.youtube.com/watch?v=YGIvWT-NBHk&feature=emb_logo

The talk is from 2018 and references some of the trends in the industry towards more automated testing.

ehnto2• 4 years ago

Fantastic games. Also, multiplatform sounds like a nightmare to manage.

0xCMP• 4 years ago

This kind of reminds me of making UIs in SwiftUI which requires a way to render in the editor so you need to provide test data and I thought it was neat because it sort of acts like a test.

jasonjmcghee2• 4 years ago

Really enjoyable read!

A small note, the code samples are a rather large font size on mobile. Setting the size to 0.9em for mobile widths works well for me!

ChadNauseam4• 4 years ago

Thank you! It doesn't look like that when you make the window small in Chrome so I didn't catch it.

jasonjmcghee2• 4 years ago

You can use dev tools to simulate various mobile devices!

https://developer.chrome.com/docs/devtools/device-mode/

harles3• 4 years ago

Based on many of the comments here, it seems like there’s a lot of misunderstanding of what an ECS is - unfortunately the name does this no favors as it should probably be written as entity/component/system paradigm or renamed entirely. It’s not just a system that contains entities and components (that’s not what “system” refers to in the ECS name).

ECS is a paradigm where entities are the data-free IDs, components contain data associated with entities, and the systems operate on entities and components in batch, usually with database-like queries.

I have yet to find a good ECS reference, but Unity’s intro[0] gives some decent flavor for it. Note that only the DOTS subset of Unity (part of it anyways) uses an ECS. The default GameObject/MonoBehavior way of writing code in Unity isn’t part of an ECS.

[0] https://learn.unity.com/tutorial/entity-component-system

Buttons840• 4 years ago

So your saying everything is a entity, and that entities have components, and the systems operate on entities (possibly based on their components). Do I understand that correctly? That understanding is definitely worth the time it took to read.

pmichaud• 4 years ago

This is essentially correct, yes.

reidjs• 4 years ago

That’s my understanding too. Think of super Mario.

Entity is Mario, the koopa troopers, the coins, the clouds, the floor tiles, the spikes, the mushrooms. All the proper nouns.

Component is the XY location in the world, ability to jump, sprite image, the height, are they affected by gravity. All the adjectives.

System is gravity - all entities with weight drop at the same rate until they hit ground. Another system may be the collision mechanic - if a mario entity lands on a koopa entity, the Koopa dies and some sound effects plays.

pizza2342• 4 years ago

is a entity, and that entities have components, and the systems operate on entities (possibly based on their components)

In an object-based design, objects are monoliths, with fields. You get a reference to player, and each Player instance has attributes. When you work on a player, you get the Player instances as a whole.

In an ECS design, you'd create a player entity; the main differences are that the Entity is just a reference, but most importantly, you access each field (called "component") separately.

The bulk of querying is performed on components, which, importantly are all global; entities are typically retrieved starting from a component instance they own. This makes the radical difference with a traditional object-based design; example:

- in OB, you have, say, players and enemies; when you draw, you'll cycle the players, and enemies, and draw them.

- in ECS instead, you'll query something like "drawable sprite" components, and draw them

It's a very interesting design, as it forces to reason more generically. It tends to get messy very easily though, since it's harder to think in terms of location of a system (in the project).

The best book for beginners to understand ECS is (IMO) Hands-on Rust (https://pragprog.com/titles/hwrust/hands-on-rust).

So your saying everything is a entity

Not sure about other ECSs, but Bevy has also the concept of "resources", which are globals; they're monolithic, and there's only one isntance of each type.

Nition• 4 years ago

It gets particularly confusing because the standard Unity paradigm is still one of entities and components, as I believe was first promoted by Dungeon Siege at GDC 2002[0]. But it's not quite the same as what is called an ECS.

[0] https://www.gamedevs.org/uploads/data-driven-game-object-sys...

KronisLV2• 4 years ago

We could all probably benefit from some slightly better naming, so that EC and ECS don't refer to almost completely different things. Otherwise you get some of the confusion as Java and JavaScript might have had back in the day. If the concepts are so different, they should be named more distinctly.

andrewflnr2• 4 years ago

This Rust conference talk, of all things, is still my favorite ECS explanation, because she builds the concept piece by piece starting from megaclass-hell to full entities-as-IDs with reasons for each step. Also she's a pretty entertaining speaker IMO. https://youtu.be/aKLntZcp27M

fartsucker69• 4 years ago

This is also one of my favorite talks, because it showed me that Rust is mostly recommended by people who don't know what they're talking about made me not waste any time on it (besides the horrendous compile times in real projects even compared to cpp).

For example, she spends a significant amount of time there explaining how to implement her component placement strategy while working around the borrow checker and at the end proclaiming how great it now is compared to a cpp implementation...

...not realizing that what she's just done is implement a custom allocator (not literally allocating memory, but the operations are the same: give me space for a component, free it, memory is latter accessed by an address instead of being some opaque thing hidden behind language semantics, etc) that can fail in most of the typical ways and that she's gained nothing over the cpp implementation while wasting effort to force Rust into allowing her to do something really basic.

dkbrk• 4 years ago

She also wrote a long-form essay which covers the same material but goes into much more detail: https://kyren.github.io/2018/09/14/rustconf-talk.html

bodge50002• 4 years ago

I think maybe the problem with explaining ECS is that the database-like part you mention is just given a passing mention, rather than being a critical part of the explanation. Obviously ECS is not a database which I think is why people avoid it, but it really helps for understanding it.

Entities are basically tables (just the table, not the data in the table), components are fields/data in them and systems are the SQL you write for them. Its obviously not a perfect representation, especially for components, but it gets the rough idea out there.

Maybe CSE is a better name for it. Its a longshot, but maybe entity being first in the list tricks people into thinking they play a big part, whereas actually they are, as you say, just an ID. Components are probably most important, then systems, then entities

Negitivefrags• 4 years ago

I would say Systems are Tables, Components are rows, and Entities are Primary Keys.

charcircuit• 4 years ago

the benefit of automated testing is that it scales in O(1) with respect to number of code changes, while playtesting is O(n)

This isn't true. Running more tests takes more time even with automated tests.

ChadNauseam4• 4 years ago

True, although the constant factor can be pretty low depending on how many cores you have :P

evgpbfhnr• 4 years ago

I never understand why people always seem to think 'test flakiness' should/can be fixed by saying "it's OK if test works 1/10 times" -- if it's so dependent on the RNG then something is bound to fail for users as well, so either the test is not representative of the game or something is rotten.

I guess time to fix tests is expensive...

jamesfinlayson2• 4 years ago

Agreed - at a previous job lots of time was invested in setting up test retries, and splitting test runs out so that you could retrigger only the subsets that failed instead of the whole test suite.

ChadNauseam4• 4 years ago

That's a really good point. It would be interesting to run the tests 10 times and have the CI fail if it doesn't pass all 10 times.

pydry• 4 years ago

It depends entirely on the reason for that 1/10 times.

E.g. I had a test that hit a 3rd party sandbox that would go down occasionally. I could have mocked it to avoid this but I had way bigger fish to fry. The flakiness triggered a very specific error that we felt safe ignoring (other kinds of flakiness we would not ignore). The corresponding prod API never had issues.

Sometimes it's a select statement without an order by in which case even if it's not strictly a bug but just quickly adding an order by solves the flakiness forever so you might as well.

Sometimes it's a pretty terrifying race condition in the code that will absolutely crop up in prod.

Operyl• 4 years ago

The only big-budget one I know of that has publicly put a large effort into it is Sea of Thieves.

Another big big title that makes use of automated testing is Riot’s League of Legends.

https://technology.riotgames.com/news/automated-testing-leag...

jerf• 4 years ago

I have often thought that GUIs, including browsers, are hard to test because the monstrous frameworks that they are make no accommodation for testing, so... of course they're hard to test. Not only are you writing a test harness around your own code, you also must rather laboriously and unreliably harness up their code too, and from the "outside" not designed to be used the way you're trying to use it to boot. Now, I don't necessarily fault GUI designers for not thinking about how to expose testability easily to their users, they're big enough projects as it is, enough to choke even a major multibillion dollar corporation, but it doesn't mean I'm any less frustrated while using them.

Game engines strike me as much the same thing for similar reasons. Yes, it would be really hard to not only have Unity as it stands today but also have all the requisite changes it would take to make it really testable, but without that you're really going uphill. It's an uphill battle at best and a sheer cliff face at worst trying to test these things. And of course when that's the case, every step your local engine programmer takes only takes you farther in the direction of being untestable because why would they bother structuring their code to be testable when the framework it is embedded in is untestable?

I have no solution. Only the observation that both of these systems (and also the browser, if you think of "the browser" as something other than a really funky large GUI) have problems from the very, very beginning, and until the foundation gets its act together w.r.t. testability it is always going to be a major uphill battle with anything built on that foundation.

(I'm not so idealistic that I imagine any world where a AAA game has 100% automated test coverage or anything, but even in my own limited experience in game dev and less limited, but still limited, experience in GUI programming, both places where I tried to leverage my extensive years of writing automated tests even in hostile legacy codebases, I found myself not merely stymied a bit, not merely inconvenienced a bit, but comprehensively blocked at every turn. An simple example: A table cell in a table widget that had a .SetColor, but no way whatsoever to fetch the color. Meaning my code to highlight rows matching a certain criterion could not be tested; in this case, for that reason alone. What a ^#&*$@'ing stupid reason to be unable to test some non-trivial code, at least from my perspective gained over years of writing a lot of other test code... and discovering plenty of errors in code of similar complexity in places where I could test it.)

jamesfinlayson2• 4 years ago

For the last few years I've been trying to add unit tests to all of my personal projects. I do a bit of game modding in my spare time and those are the last projects that ended up getting unit tests, but they were also the ones I most wanted to add tests for because they're the most complicated and also the most time-consuming to test manually. The biggest hurdle to start with was that the Visual Studio projects are all set up in such a way that integrating Microsoft's C++ unit test framework either requires you to (probably) recreate the project the right way, or to hack around with scripts to create a working test runner. Setting up Boost tests might be less painful. But once that is done, my next hurdle was mocking out the game engine. Not overly difficult as the engine exposes a nice API but still time consuming. Now that that's done it's been good start adding tests, though I haven't added any tests for anything too complicated yet.

syntheweave• 4 years ago

The post contains a common fallacy that you will know of if you have tested substantially sized games:

What ECS architecture does is offload the testing burden into data instead of code. The code itself can do the right thing in all circumstances, but as you componentize your higher-level behaviors, more business logic becomes bindings of data to other data. So after a 100% green test suite, you still have bugs and all of them are of the form of "mysterious runtime behavior". The engine code doesn't know how to automatically check for a "good data shape" or flag it as a bug; you can add checks, but it's project specific and asset-specific compiler development: a missing animation is a different kind of bug from a typo in a text prompt, or a hitbox that's slightly misplaced. Thus, for a lot of categories of assets, your cheapest test is still to periodically eyeball things and say "that's in spec" or "that's fishy".

ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.

echelon• 4 years ago

ML AI might actually be able to change this scenario by acting more like human players and deriving more second-order metrics.

New startup idea. Probably applies to any "hard to test" thing.

yladiz• 4 years ago

I think you're reading something that isn't being argued in the post, though. The author is not arguing that you will have no bugs in your game if you use ECS and add tests, it's that testing in this way can make it easier for development because you reduce the possibility of introducing a bug inadvertently in one of your systems and then have to rely on QA to find it later (components too I guess, but I would guess component "tests" can be covered by types since they're just data and the systems are the ones that interact with the data). The kinds of bugs you mention are of course different and some of those kinds of bugs aren't worth the trouble of even trying to test, but if your game is shaped in a way that it is basically a collection of code modifying data that represents the game, you can eliminate the need for a lot of manual QA testing by just automating it.

xwolfi• 4 years ago

And it's not just games, this trend of moving code into data. I've been in several companies where their fundamental logic was a configuration interpreter, very complex, tied to a gigantic set of xml or json "configurations" which themselves had interdependent logic written as structure.

So when maybe the first iteration 20 years ago was all fun and giggles "look a monkey could edit a few knobs and it all moves around fine", today it's a tangled mess that an ever rotating set of new people build on top of with no validation tool, no formalized syntax, no ability to test nor explain the holistic logic of.

So, as you say, we end up running the thing and looking if the output makes sense, the code working more out of luck than engineering.

nextlevelwizard• 4 years ago

But in videogames, correctness matters barely at all.

This is pretty weird blanket statement or rather it very much depends on what kind of game you are making. If you are making a casual single player experience then having glitches isn't that big of a deal. However if you want to make the next e-sports game then glitches matter a lot more especially if they can be abused to gain power - be it actually powering up your character in some way or having (near) invincible position.

Another aspect many game devs ignore is speed running. Having glitches in a single player game might not be that big of a deal for normal playthroughs, but many games have had long lives after their initial release as people want to speed run them and there glitches matter. If your game is broken it can create situations where speed running isn't fun.

Automated testing in Bevy

Related Stories