A Thought Experiment: Using the ECS Pattern Outside of Game Engines

http://adventures.michaelfbryan.com/posts/ecs-outside-of-games/

by MichaelFBryan • 7 years ago

176 73 7 years ago

adventures.michaelfbryan.com

adamnemecek2• 7 years ago

I've been playing around with ECS. It does seem to solve a lot of architectural problems I had.

jrhurst• 7 years ago

I find ECS a pretty intuitive concept, but it always ends up being a massive yak shave to me

jayd166• 7 years ago

Agreed. The vast majority of the benefit is simply from learning to use composition over inheritance.

Most applications that aren't games don't need to support things like finding all active objects of a type or maintain many discrete actors executing their own run loops.

But Unity isn't dumping a lot of marketing into talking about the decorator pattern so everyone is jumping right to ECS now.

ratww4• 7 years ago

What were your problems with it? Was it using a particular library or engine, or you did it yourself?

I'm curious because I started using ECS-like techniques exactly because traditional OOP in games felt a lot like yak shaving.

kevingadd2• 7 years ago

When performance matters I often find myself yak shaving existing code to basically turn it into structure-of-arrays format (like is required for ECS), if not converting to ECS entirely. The alternatives scale so poorly...

netgusto• 7 years ago

I did this lightweight ECS for Go and TypeScript:

TS: https://github.com/netgusto/ecs-typescript

Go: https://github.com/bytearena/ecs

Very interesting programming concept indeed!

---

Edit: These implementations support tagged queries, and indexed views for faster iterations (ECS uses iterations a lot).

zubairq• 7 years ago

How is an Entity Component System different from Objects and Controls in something like Visual Basic for example?

adamnemecek2• 7 years ago

Ecs is more like a database.

TeMPOraL10• 7 years ago

ECS is all these things and none of these things. It's several separate concepts under the same name; different applications/games pick different pieces of it.

It's kind of meta, really; you can imagine ECS itself as an entity, and different meanings of it as different components that could be included in it.

m12k• 7 years ago

I wrote a comment on another post where I explain the cache optimization benefits of ECS, https://news.ycombinator.com/item?id=21674818

ratww4• 7 years ago

Visual Basic controls are more like React components, I think.

ECS is, traditionally, just a way to dynamically add behaviour (components) into "blank slate" entities.

Instead of making a EvilMonster class in a game you just have it as a conceptual entity (normally it's just an integer, like a database id) associated with a bunch of components (normally just plain structs) and their parameters: Renderable, RigidBody, Collider, HasEnergy, HasTransform, AudioEmitter, etc.

You can stitch entities and components together using either code or some external data source. The "S" part of ECS is called the system, and it implements the behaviour defined by components.

The upsides of this IMO is that this is a fantastic way to structure your code. The data-oriented aspect is also great for enabling non-coders to assembly complex entities. And some other clever people (such as the sibling answer) found out this is also a great optimisation technique.

There are multiple implementations, of course, so people have different opinions.

aliswe3• 7 years ago

I'm accidentally using this approach in my CMS for saving .NET Core classes (with interfaces - and supports explicit interface implementations) into document stores.

EDIT: Actually, now when I think about it, this approach sorta makes every object (or Entity) into an independent, isolated database. Hm ...

TeMPOraL10• 7 years ago

Which ECS? :).

Despite "clear" definition from Wikipedia, ECS as a pattern suffers from multiple personality disorder. There are several different goals that all share the same name, and different implementations pick a different goal set, ending up looking not quite the same, and getting different kinds of benefits.

So outside games, if you pick "ECS as composition over inheritance", you get something like here, or Clojure's "let's use maps for everything". If you go for "ECS as a performance optimization" - a frequently touted benefit - you'll end up storing a lot of global arrays (perhaps arranged in structures), each packing every instance of a component's property across all entities.

If you go for "ECS as a way to have data-driven entities modifiable on the fly" or "ECS as a way to make game logic cleaner", then congratulations, you've just reinvented a relational database! Since you're not developing a game, you might as well store your data in SQLite and call it a day.

(In fact, I'm currently working on a roguelike game as a side project, whose distinct feature is that it stores all its game data in an in-memory SQLite database, precisely to experiment with a pure form of "ECS for game logic" pattern.)

In the end, the author explored one way of doing ECS. There are plenty others, worth their own thought experiments :).

codetrotter• 7 years ago

If you go for "ECS as a performance optimization" - a frequently touted benefit - you'll end up storing a lot of global arrays (perhaps arranged in structures), each packing every instance of a component's property across all entities.

This is basically what I am doing in the backend I am writing for an app that I am working on. (Also not a game btw.)

No database, no overhead :D

You really can fit a lot of data in memory, and I think a lot of people tend to forget that.

I so so wish that I get to see the day when memristors become cheap and abundant. It would be so pleasant to program for a system like that I think. No persisting to disk or loading from disk, ever – imagine that!

BubRoss3• 7 years ago

New NVMe drives claim 5GB/s on PCIe 4, is memory mapping something like that not good enough for you already?

imtringued2• 7 years ago

You always need a memory hierarchy. Not even DRAM is fast enough for our CPUs so we have L3 caches and those are not fast enough either so we also have L2 and L1 caches and registers and physical wires which temporarily store the state of the processor by taking advantage of propagation delays.

ratww4• 7 years ago

Not that different from MVC a couple decades ago, then! :D (just kidding)

I love the history of ECS. Scott Bilas' goals with his work in Dugeon Siege, and before that, in Gabriel Knight 3 [1], was to have something close to what you're doing, in order to speed up development. He was storing game logic in text files, as opposite to having it hardcoded, to enable level designers and artists to iterate faster. I guess he would have considered SQLite if it was as widespread back then.

It was only later that people realised ECS could be a performance optimisation, associated it with OOP and made libraries for it.

[1] A very fun adventure game, and fun to reverse engineer. One of the "Easter Eggs" involves teaching you how to extract data from itself.

TeMPOraL10• 7 years ago

Didn't know that origin story, thanks!

FWIW, what I'm doing is storing all the runtime state in in-memory SQLite, instead of writing my own ECS. Components are rows in tables, entity IDs are used as foreign keys, game logic just does regular SQL SELECTs to query the game state, and then UPDATEs, INSERTs or DELETEs stuff as needed.

I came up with this experiment when I was writing yet another implementation of ECS for that game, and in the middle of optimizing it for processing performance I realized that all I'm doing is essentially hand-writing database indexes, so why not use a proper database instead? And it turns out, SQLite is fast enough when used in-memory.

BubRoss3• 7 years ago

I have gone down the exact same line of thinking. I think there is value in creating a simple relational data structure to have simple select and insert functions but not the overhead of sqlite. Many times when I think about it, the state of most programs could use a handful of the same structure and be done.

TeMPOraL10• 7 years ago

Yeah, I think so too. My goal after hopefully making a completely playable game running on SQLite is to eventually rip it out and replace with my own ECS, based on gained experience. I start with explicit SQL queries, and am slowly building an interface on top of it. Or rather, I'm moving towards the interface I abandoned prior to switching to SQL.

Before SQLite, I was working on my (yet another) own implementation, and eventually realized that I don't like the "Systems" part of the ECS, given I planned a lot of logic that required inspecting multiple components of any given entity. Thinking about it, I figured out that a better abstraction for a "System" would be a piece of code executed on a set of entities, defined by some selection logic. That is, a query. I quickly ended up with this interface:

  (select-entities 
     :components (has-position has-health (not player))
     :where ((entity-id (has-health (hit-points hp)))
             (< hit-points 50))
     :order-by (((lhs (has-health (lhs-hp hp)) &optional (targetable (lhs-priority priority)))
                 (rhs (has-health (rhs-hp hp)) &optional (targetable (rhs-priority priority))))
                (if (and lhs-priority
                         rhs-priority
                         (= lhs-hp rhs-hp))
                    (< lhs-priority rhs-priority)
                    (< lhs-hp rhs-hp)))
     :into my-array)

(The example selects non-player entities with position and health components which have less than 50 HP, and orders them by HP and, if both compared entities have a "targetable" component, also by targeting priority.)

Which, as you can see, looks essentially like an SQL query. :components is select + join, :where is filtering condition, :order-by saves me from doing an explicit sort on the results, and the :into my-array part allows this query to not allocate new storage for results each frame.

Now each time such call showed up in my code, some extra book keeping storage would be allocated, and underlying logic ensured that most of the query doesn't have to be recomputed every frame. For instance, the :where part was handled by tracking addition and deletion of components to/from an entity. Implementing similar optimizations for :order-by part, I quickly realized that most of my "book keeping" tricks are essentially equivalent to compiled queries and indices on tables. At which point I ditched it all and switched to SQLite.

(I meant to write a blog post about this at some point. I guess I just wrote half of it here.)

imtringued2• 7 years ago

Last time I built an ECS it was effectively a bare bones relational database. A system just queries the information that it needs. The code basically looked like this. There was no need for explicit where or select queries because all I really needed to check was the presence of a component. Here is a contrived example. ecs has an inner join function that accepts all components (stored in an array) that you want to join and everything gets passed to a callback that receives all the components.

    void autoheal_system() {
         ecs.inner_join(ecs.status, ecs.inventory, ..., [](status, inventory, ...){
             if(status.health < (0.2 * status.max_health) {
                inventory.find("potion").use()
             }
         })
    }

One big advantage with this system based implementation is that you are processing all entities in batches compared to the naive OOP style where executing entity.foo() may run Dog.foo or Cat.foo and thrash the instruction cache and cause branch misprediction because of the virtual function dispatch that is constantly switching between functions. I haven't done parallelism with this pattern yet but as long as you do not cross reference other entities it shouldn't be too difficult to just slap on an OpenMP pragma and have a parallel_inner_join function that runs everything on multiple cores.

aaron-santos• 7 years ago

or Clojure's "let's use maps for everything".

Small world. I'm writing a roguelike in Clojure and indeed use maps for everything. Datascript seemed promising, but it is hard to beat Clojure's builtin data selection functions.

TeMPOraL10• 7 years ago

Small world indeed. I work a bit in Clojure from time to time, so I have a passing familiarity with the coding style.

FWIW, for the same game I described above, I also ended up cloning half of Stuart Sierra's famous Component library to use it as my half-baked Dependency Injection framework. This was motivated by the fact that - due to various life reasons - I do a good chunk of my development from a sidearm machine via SSH to my desktop, so I need an architecture where I can develop and test the game in headless mode (no graphics, mock inputs) as much as possible. I write a lot of unit tests now.

Razengan4• 7 years ago

I could use some critical feedback on my system [0]. I went with what felt to be the most intuitive pattern for my own brain, but later found that it was apparently against the grain: making everything plain old classes and keeping all the logic inside components.

When I need a particular behavior, I just create a new class and keep everything related to that behavior in a single file. Entities and systems are little more than arrays, and components can query their entity for other components at runtime.

So there may be a MouseEventComponent or a TouchEventComponent, added by the engine depending on your device, then a PointerEventComponent to provide a layer of abstraction for OS-independent player input, and finally some PointerControlledShootingComponent etc.

Or a KeyboardEventComponent feeding a generic DirectionEventComponent, and you can swap the keyboard for a gamepad at runtime, or have different input sources for different entities.

To my understanding, doing stuff like that did not seem as intuitive in the more "traditional" ECS patterns, as opposed to just editing a list of classes to update every frame, or components holding references to other components in other entities (e.g. to share an input stream, or to implement a self-playing demo mode.)

It's meant to be for 2D games only and I haven't hit a wall with it so far, performance or otherwise, but some things like serializing the object graph for save/load or parallel execution might be a little tricky when I get to those.

My goal is to eventually be able to describe game scenes with a declarative SwiftUI-like syntax, where you'd add components like view modifiers and so on. Come to think of it, SwiftUI might be considered as an ECS for apps, or is that stretching the definition a little too far. :)

[0] https://github.com/InvadingOctopus/octopuskit

TeMPOraL10• 7 years ago

That's an interesting take. I read the README and the architecture document at [0]. Based on that, the way I see it, your approach is different but essentially equivalent to the typical non-performance-oriented ECS architecture. Entities are still runtime-configurable, Component updates are done in batches and in appropriate order. You've thinned out Systems, moved their code to their corresponding Components, and provided places to hook component mount/unmount code.

To me, doing stuff like this seems more intuitive than more "traditional" ECS approaches, but perhaps that's because I was never comfortable with the idea of Systems.

Eyeballing the code, the only slight architectural drawback I see is that the code must always belong to a particular Component, even when that code operates on multiple Components as equal partners (this is the ECS eqiuvalent of "your language doens't have multimethods" problem in OOP). If I understood correctly, you solve that by creating a Component composing other Components.

One particular thing I like about your overall architecture is that it explicitly covers state machines - both at the game (scene) level and at per-entity level. Various other ECS examples I saw on the web tend to completely ignore this topic, and it's pretty much the first thing I hit when I start using them: where do I put state machines?

Overall, OctopusKit looks like a solid piece of work. Congratulations, and do a Show HN: if you haven't done it yet!

[0] https://invadingoctopus.io/octopuskit/documentation/architec....

jayd166• 7 years ago

Isn't this usually referred to as the decorator pattern when used this way outside of a game loop?

golergka2• 7 years ago

The critical differences are memory layout and explicit declaration of logical dependencies: this makes the code parallel by default.

jayd166• 7 years ago

This is just false unless you think the featured article isn't ECS. Rendering is an ordered operation that at the very least shouldn't be considered parallel by default. Simply putting it in an ECS pattern doesn't guarantee anything like that.

Its probably better to say it might help you write tighter loops because (ideally) you have a small amount of code looping over a large array of data, instead of a sea of actors hopping around the heap. Of course, you can make tight loops in a lot of different patterns...

akhosravian• 7 years ago

Short answer is no. Decorator is an OOP design pattern, ECS is an architectural pattern. In that sense it’s a more abstract concept. One could say decorator is an OOP form of it, but I think the data oriented design crowd would balk at doing so in a mainstream OOP language like Java or C#.

jayd166• 7 years ago

I suppose you're right. ECS explicitly calls out a system that separate from the components while most of the decorator examples run code in the components. I've written decorators that are closer to ECS but I guess its fair to say ECS implies a specific implementation.

k__• 7 years ago

I think a decorator could be implemented as ECS.

You decorate your entities with components that get updated via a system.

golergka2• 7 years ago

That's exactly the idea that I have been toying with recently: using ECS (and even spec crate in particular) to implement an authoritative game server. Haven't yet figured out how to make spec's and tokio's scheduling to play along, though.

JoshMcguigan• 7 years ago

I look forward to seeing the ladder logic editor. I run PLCFiddle [0], which is a purely React ladder logic editor.

[0] https://www.plcfiddle.com

fulafel4• 7 years ago

In the Clojure world, this used to be a popular way to structure backend apps: https://github.com/stuartsierra/component

Components provide some basic guidance for structuring a Clojure application, with boundaries between different parts of a system. Components offer some encapsulation, in the sense of grouping together related entities.
Each component receives references only to the things it needs, avoiding unnecessary shared state. Instead of reaching through multiple levels of nested maps, a component can have everything it needs at most one map lookup away.

TeMPOraL10• 7 years ago

Used to be? What replaced it? I used it in my last Clojure project, and I'm about to start expanding it further; should I replace it with something else?

fulafel4• 7 years ago

It's fine, hasn't just been getting much press lately.

heretoo• 7 years ago

try integrant

https://github.com/weavejester/integrant

You can move from using component's functions/macros to using a map and cross references using reader literals.

Having used both, I find integrant to work better for me.

dkersten• 7 years ago

Stuart Sierra components and ECS components are not really the same thing though. They share the name “components” but not thaaat much else.

Stuart Sierra Components are more akin to Unity’s behaviour components prior to their introduction of an ECS. They’re a way of encapsulating state which may have dependencies and has a lifecycle (needing to be started or stopped).

ECS components are purely blobs of (usually fine grained) state, the existence and combination of which in an entity creates behaviours (that is, the systems which implement the behaviour will dynamically do so for any entity that has the prerequisite components, but the components are just data).

I always preferred naming ECS components as “traits” (they are the traits that an entity has, eg “animated”) and the systems “behaviours” (they are the behaviours exhibited by entities that have certain traits).

spankalee3• 7 years ago

So many of these kinds of OOP modeling complaints and solved by mixins.

_carl_jung• 7 years ago

I'd earnestly like to hear the responses to this from the down-voters. I had the same thought and suspect there is a good reason not to use mixins.

One problem off the top of my head is run-time changing of components. An entity that inherits from multiple mixins can't inherit from new mixins (or lose existing ones) at runtime.

skybrian2• 7 years ago

Yes, ECS is usually about runtime changes in behavior. It also uses IDs rather than hard references, like in a database. Deleting an object makes references to it invalid, rather than references keeping objects alive like in a functional or object-oriented object graph.

aliswe3• 7 years ago

Are you saying that this is different from mixins?

spankalee3• 7 years ago

Absolutely. Single inheritance is a degenerate form of mixin inheritance, where a subclass has a single, fixed superclass.

Mixins simply fix single inheritance so a subclass's parent class is parameterized, and so concrete classes can be composed of several non-related mixins (which will often seem like multiple super-classes).

ECS is a much different and bigger beast. It's a runtime composition pattern, and has almost basically to do with classes or objects.

If you're only complaint about OOP is with single-inheritance, ECS is probably overkill.

raverbashing• 7 years ago

This sounds a lot like OOP purists finding out not everything can be solved by inheritance so they rename something that existed already.

5cott02• 7 years ago

Apple provides an ECS in gameplaykit that can be used in iOS/UIKit apps, not sure why anyone would want to though.

https://developer.apple.com/documentation/gameplaykit

Razengan4• 7 years ago

not sure why anyone would want to though.

What do you mean by that?

GameplayKit has its limitations, but I've been building an entire engine around it, and I like it so far, especially for the ability to stick with pure Swift and native APIs:

https://github.com/InvadingOctopus/octopuskit

5cott02• 7 years ago

Not sure why anyone would want to use GameplayKit ECS as a replacement for MVC/MVVM arch in a UIKit based app.

almindor• 7 years ago

I've built the Texel ASCII Art Editor (https://crates.io/crates/texel) using Specs (https://crates.io/crates/specs) as my first experience with ECS.

The original goal was an ASCII game but since I needed to create the resources for it it morphed into Texel. I think the use of ECS here wasn't a bad decision but Specs proved to be just too cumbersome and overoptimizing.

As others have mentioned ECS is a bit "loosly defined" and each implementation seems to go over the line to add more specializing one way or the other. I want to switch to something more simple and elegant like DCES (https://crates.io/crates/dces) for my next refactor.

I think for "runtime resource management" ECS is fine if you need a fairly large, distinct pool of entities to handle. One of my main usage problems was the re-use of same components in the same entity.

E.g. imagine having an entity with a global world position, but also an internal position for something like "last cursor position". It's not possible to just "add another Position component" to the same entity, for good internal reasons, but still. It's pretty important people take these kind of limitations into account before planning out their entity/component maps.

nabla9• 7 years ago

I have been using this.

Common Lisp Object system perfect and natural for this kind of programming.

shoo• 7 years ago

See also: Richard Fabian's "Data-Oriented Design" book. The fourth chapter is about component-based objects, and the preceding chapters about relational databases and existential processing are relevant.

http://www.dataorienteddesign.com/dodbook/

Prior HN discussion: https://news.ycombinator.com/item?id=20380397

ncmncm• 7 years ago

Who are these people who think that inheritance is the solution to every architectural problem, or any?

I didn't think that Java coders were especially well represented in the gaming world. It has been over 20 years since C++ programmers learned that inheritance is just a way to dress up function pointers (not that that does no good, but function pointers are pretty specialized machinery).

Seriously, you don't need to use a named style to write programs. Your language offers lots of different facilities you can put together in any way that is useful to achieve your aims. Functional, OO, ECS, MVC, procedural, data-oriented, reactive, whatever -- it is all just code.

You have only three problems: code needs to be organized well enough that you can understand it, code should run fast enough, and you should be able to change one thing without rewriting everything else. Nothing else matters.

A Thought Experiment: Using the ECS Pattern Outside of Game Engines

Related Stories