Game engines are more than libraries glued together

https://www.leafwing-studios.com/blog/game-engine-glue/

103 97 4 years ago

KronisLV2• 4 years ago

It gets so much worse though. Suppose you need that bug fix from your dependency, or worse, want a shiny new feature to unblock your work?
Roll a d12 to determine what happens:
... (a lot of examples of no attention being given to the issue)

Personally, I'm all for open source, but to me this seems inevitable when you expect people to solve your requests and issues with (quite possibly) no compensation.

That's a situation where paid support of any sort with SLAs would help mitigate almost all of these things. Similarly, if you couldn't get a contract from the current maintainers of the project (or couldn't find those maintainers in the first placE) then it would be telling, that you should probably look elsewhere.

Then again, curiously, open source projects rarely if ever offer paid support and using closed source libraries comes with a wholly different set of risks.

It does seem like it would be a reasonably lucrative area, though: "Pay us X$ per month to have your issues reviewed within Y hours." Or maybe even just allow the interested parties to donate money towards the resolution of particular issues that matter to them: "So far Z$ has been donated towards the solution of this particular issue. This issue has the #1 highest amount of donations and should be given priority."

anonymoushn2• 4 years ago

Interestingly "You patch the issue and ship the patched library in your game" is not on the list.

Comparing to popular game engine Unity, let's say you are making a game that uses time-travel dead-reckoning netcode and polls inputs at 60hz. So you'd like to poll inputs at 60hz even if the game is rendering at 20hz, and you'd like to send those inputs in UDP packets immediately, not up to 50ms later than you got them. You don't need to roll a d12, we already know what happens: you start waiting in June 2005 for the ability to poll inputs at 60hz while rendering the game at some other rate, occasionally posting about this issue on the official forums. You continue waiting for at least *17 years*. It's a good thing you didn't depend on an open-source library instead!

dkersten2• 4 years ago

Isn’t Unity’s code available? Unreal’s is, at least. It’s not quite the same as having official support and you don’t have the same rights as in open source, but you can at least change the code and fix it yourself if you really need to (with the difficulty of understanding a complex codebase, but that’s the case in open source engines too)

ratww20• 4 years ago

Sorta unrelated, but I see the existence of engines to also be sort of a problem for Open Source libraries too. They create sort of a maintenance bottleneck.

In other areas where open source is more prevalent (like web dev), there will be virtually a guarantee that someone running into some long-lived issue within a library will try to patch and contribute back, or in the worst case fork it. Because the end users of frameworks/libraries often have the skills needed to patch those libraries.

In Games, however, there's fewer and fewer true consumer of libraries, since everything go through engines, and there's a lot of consolidation going on. Also, the popular engines aren't targeted at programmers that could fix the libraries, quite the opposite.

That puts a lot of pressure on library maintainers and engine maintainers, as they become the only people able to fix those deeper issues. Bevy of course is a bit lower-level, an others have source-available, but still...

robalni• 4 years ago

Personally, I'm all for open source, but to me this seems inevitable when you expect people to solve your requests and issues with (quite possibly) no compensation.

Let's remember that half the point of open source/free software is that you have the freedom to fix things yourself. Remember that the reason that Stallman started his free software work was that he couldn't fix a printer because he was dependent on a company that wouldn't give him the source code.

I don't understand how we came to this idea that open source/free software means that people are dependent on each other to fix each other's bugs when the point of it was to get rid of that dependency.

jbluepolarbear• 4 years ago

As a gamedev I don’t have time to fix every issue that comes up with the engine. I’m not interested in building a game engine, I want to make games. Unity and Unreal are mature enough that they have understood limitation, workarounds of known bugs, and many, many solutions for common game mechanics. I built game engines for the first half of my career (about 6 years) and it’s a lot of work that could be better spent designing and implementing new game features. This is also why a lot AA and AAA game development are moving to Unreal and Unity because they want to make games not engines.

flohofwoe6• 4 years ago

I wish the term "game engine" would be better defined. Somehow this now means "the whole package", an integrated product like Unity, UE or Godot which includes all tooling required to create and publish games. This isn't an engine, it's a whole "Gamedev IDE", in spirit much closer to Photoshop than ImageMagick. Or as car analogy: nobody calls a whole car an "engine", even though every car has one.

The original meaning of "game engine" which made much more sense was: "the actual code which enables the game to run", e.g. just the runtime parts, the tiny piece of code that's dangling off the end of the asset pipeline ("tiny" relative to the work that needs to go into authoring tools and asset pipeline).

Let's begin with something controversial: a stack made for a single game is not a game engine.

...controversial indeed :D

ratww20• 4 years ago

IMO this coupling happens mostly on purpose. It's just vertical integration, Apple style. It seems to be in the interest of "engine" makers to make people think that a smaller engine without an editor can't succeed, and that there's no possibility of having a standalone editor. They want control of the whole stack.

Game development in general has been plagued by those things. Everything uses proprietary formats, little interoperability, lots of custom stuff. All the parts that aren't proprietary came from the outside (eg: audio/video/image formats) or was leaked by generous developers (Quake MD? and BSP formats being used in newer games/libraries).

I would personally love to see more people working on editors that can be used by multiple engines, so that we can see more focused engines grow too, without having to create a whole editor ecosystem. And also editors would grow themselves, and editor-makers would make one without having to make a full engine. I really think it's possible.

...controversial indeed :D

I think the article writer is flat out wrong on this one. The Doom and Quake engines are definitely... well, engines, even though they were only reused later. Sure you can have a more spaghetti "game without engine", but I'd draw the line whether it's reusable/logically separated or not.

flohofwoe6• 4 years ago

100% agree. A popular "bring your own engine" editor would be awesome and would make a lot of sense (just not a lot of 'business sense').

The Unity editor is actually quite hackable and can be used like that, but that's very likely a 'licensing grey area'.

Jare• 4 years ago

It seems to be in the interest of "engine" makers to make people think that a smaller engine without an editor can't succeed

Phaser for a long time was a runtime engine without an editor, and many people missed that type of capability. Same with Cocos2D.

and that there's no possibility of having a standalone editor.

Tiled (for 2d maps) or Spine (for 2d animations) are 2 examples of well known standalone editors. There's no chance to "make people think" that such things can't exist.

It seems to be in the interest of "engine" makers to make people think

There's no hidden agenda going on, it's that integrated tools are often much more effective when building things out of multiple different pieces. Wouldn't it be cool to place animated objects in Tiled and see their animation states to properly gauge how they interact with the rest of the scenery in different states? Or use Spine to animate a tilemap going through an earthquake?

Custom file formats exist because features differ, and even for similar features, the semantics differ.

bob10292• 4 years ago

Decoupling the user interface from the actual engine seems like an excellent idea on the surface.

Aside from your post here, has anyone even proposed an "engine agnostic" IDE experience for game dev? I can kinda see how it would be possible to do this - code your engine against some common editor API rather than the other way around.

The idea of being able to swap to an entirely different engine but keep all my content, tools, etc is pretty fantastical, but I don't see why it would be impossible.

ohgodplsno5• 4 years ago

It's not just vertical integration for the sake of it. There are many, many, many things that are simply made easier by having every component able to talk to eachother. UE5's particle system can talk to the sound system back and forth, which can trigger gameplay events and send data to the renderer, etc. This leads to extremely complex interactions being rather easy to do, and that is exactly what large games want nowadays. You don't want to painstakingly time your cutscene to begin exactly 2m15s after the beginning of the level because you know that's where the song will begin but shit wait the intro can loop, no, you just have a gameplay event that starts the sound, the sound regularly says "hey, if you're ready, please trigger the cutscene now" and everything works together.

Getting this with a bunch of libs glued together is simply impossible. This is a design that was thought of for years before it happened properly.

oneoff786• 4 years ago

While true: updategame() Renderframe()

pjmlp3• 4 years ago

I have books that already refer to game engine as integrated experience that date back to MS-DOS days.

mysterydip2• 4 years ago

Care to list some of them? I love collecting DOS-era game development books.

k__• 4 years ago

Bit like with frontend frameworks.

stonewareslord• 4 years ago

I agree with your assessment.

a stack made for a single game is not a game engine.

For a concrete example: the newest Lego game, Star Wars the Complete Saga, had a complete engine rewrite ("NTT") and the new game is the only one that runs it. I wonder what term the author would use to describe NTT, which isn't designed to run a single game or have a community.

ohgodplsno5• 4 years ago

What does a complete engine rewrite mean ? Is it just a new renderer with deferred shading ? Is it a bunch of new gameplay tools ? Is it a start-from-scratch thing ? (Which would be absolutely stupid).

Game engines are more of an ecosystem. Yes, the community is a thing because the devs seem to be a small indie group that uses Godot so they can ask for assistance, but Frostbite does not have such a thing (well, maybe internally at EA), and yet noone would say it's not an engine. As long as, through an upgrade, your engine keeps the same way of working and you're familiar with it, it is an engine as a whole. UE and Unity are two different engines because the workflows are different. But UE4.0 to UE5.0 is still the same engine, you won't take that much longer to get back in the groove.

otikik• 4 years ago

I always picture it like this:

The shotgun is a library. The motorbike is a framework. The (unskinned, inactive) T-100 is a game engine.

The Arnold-skinned Terminator shooting at the T-1000 with the shotgun while riding away in the motorbike is a video game.

charcircuit• 4 years ago

I think you are confused. Editors are not part of the game engine. Unity editor is separate from Unity the game engine. When you ship your Unity game it doesn't bundle the engine with it.

ratww20• 4 years ago

When you ship your Unity game it doesn't bundle the engine with it.

Sounds like you're confused yourself :)

You probably mean "it doesn't bundle the _editor_ with it".

The part uncontroversially accepted as being the "engine" is definitely bundled.

flohofwoe6• 4 years ago

Well that's how I see it too, but for most people "Unity" and "Unity engine" is the same thing.

nineteen999• 4 years ago

The original meaning of "game engine" which made much more sense was: "the actual code which enables the game to run", e.g. just the runtime parts, the tiny piece of code that's dangling off the end of the asset pipeline ("tiny" relative to the work that needs to go into authoring tools and asset pipeline).

It still means that. When you hit the "Build" button in the Unreal IDE, you get your assets and custom code packaged together with the engine code. It doesn't bundle the IDE. I haven't used Unity in years, but from what I remember it works the same.

The IDE is only there (at least in Unreal's case) to give artists and programmers an easy workflow to develop the finished product, after which it's cast adrift. You need an integrated IDE these days because the number of features that are included in the engine is insane. If you're doing shaders and you're not writing them in raw GLSL, then the Unreal materials interface in the IDE is indispensable. If you're doing visual scripting (eg. Blueprint) the same applies.

giancarlostoro• 4 years ago

To me a game engine is anything that doesn't include a full blown IDE but still allows you to make games using any IDE, closer to a library. I agree with you, I don't know what could fix this to make a subtle distinction between the fully integrated game development environment maybe IGDE? Not sure, but it is more helpful to know how much they hold your hand. I'm okay with a very small library that doesn't provide all tooling as long as its in a language with good tooling, or they provide SOME tooling.

bitwize• 4 years ago

Yeah, back when I was growing up and Carmack was everyone's hero, "game engine" basically meant the game's main loop, and everything that supported that main loop. The code that makes the game actually work.

These days "game engine" basically means Microsoft Access for gamedev. And if you don't use one you're basically leaving money on the table, so it makes no sense to try and be a Carmack in today's scene.

seba_dos16• 4 years ago

My own engine used to be a single-game stack. Now there are more than 40 games written with it out there (and only one of them with its code written by someone else than me). It's not an IDE at all, just a bunch of (often intentionally leaky) abstractions on top of existing libraries. It comes with a tiny external editor written in PyQt that produces animations that can be consumed by the engine (which is optional, as they can be created by hand too), plus a bunch of scripts that use Docker containers to effortlessly cross-build it to several platforms.

I really don't care about other people using this engine. It's not for them, even though it's public and FLOSS. It's a tool for me (and people I work with) to make it easier for me to write my games the way I want to write them.

I wonder what the article's author would call it ;]

p0nce• 4 years ago

This post hits home. It's true that if you want to make a more framework-ey library it is easy to end up maintaining a lot for controlling the developer experience. It is really sobering since it means a limit to reuse.

stephc_int13• 4 years ago

In my opinion, the definition of a game engine is the part of the code in a game that is not the content.

snarfy• 4 years ago

How are you supposed to make a level for a game if the engine doesn't support editing levels? A big part of the level is that it integrates features of the engine directly into the level, e.g. static light shadow rendering.

flohofwoe6• 4 years ago

There are so many options which people are just blind to these days because the "Unity way of doing things" has become so entrenched in people's minds :D

- you can build the level in a 3D modelling tool like 3DSMax or Maya and then write an exporter plugin to export the data into your engine-specific file format

- you can write a simple DSL to describe your level through text

- you can write a converter from another popular level file format (like Quake maps)

- you can write a plugin for the Unity editor, and create your maps in Unity

- etc etc etc... the possibilities are endless

...of course a dedicated level editing tool tailored to a specific runtime engine will always be most powerful and flexible.

chongli• 4 years ago

Or you can go down the roguelike path and generate the levels procedurally. This approach is incredibly powerful and gives traditional roguelikes a ton of replay value while being very easy to build compared to traditional static assets. Plus there are subreddits [1] [2], wikis [3], and more dedicated to the topic!

[1] https://www.reddit.com/r/roguelikedev/

[2] https://www.reddit.com/r/proceduralgeneration/

[3] http://roguebasin.com/index.php/Articles

tartoran• 4 years ago

That’s true and in general conducive to going off the beaten path and building original games. I feel that with large game engines the cost is in ubiform putput: everything looks like everything.

w4rh4wk5• 4 years ago

Build it in Blender, including cameras and lights. Then build an importer for your game. This is substantially less work than building a whole editor. Especially when considering the editing capabilities of Blender.

We kinda did that for our little ray-tracer: https://www.youtube.com/watch?v=QZSoVidqV2Q

blooalien• 4 years ago

Or build it in Blender, export it in GLTF (.glb), and import it directly into a game engine that supports GLTF import (like Godot engine for example). :)

jokoon• 4 years ago

Game engines are not good in general, because they do too many things. In my view it's better to pick libraries that do just what you need.

It gets a bit worse when you need to write a functionality that is specific to your game, and it becomes too dependent on a certain engine or library, so you can't switch to another library or engine.

Unreal and unity are monsters that allows beginners to have quick access to a high end rendering system, but then it doesn't leave enough room to add new cool things, like cool gameplay, a good multiplayer system, etc, when you want to ship on console who have limited memory.

I'm quite curious about the unreal engine and how fortnite uses it, because PUBG is a similar game, but its performances are quite bad, so it is an easy demonstration that a rich engine doesn't mean your game will be a smooth experience if your developers are not experts, because expert developer can write a better home made engine instead of paying for a rich engine.

These are the good reasons why I don't want to use all-in-one solutions, and this applies for web frameworks too. What's even worse, is that those big things don't just enforce a programming style, they also make it difficult to migrate your code elsewhere.

999900000999• 4 years ago

When you and your friends go out to get yogurt, do you generally go to a farm to pick out your cow?

The big part of what makes Unity easy to use, is just everyone uses it. I can pay an artist to make me some assets which work with HDRP. If I'm building some custom library, with custom shaders, I'm going to have to do much more work to make sure the assets are compatible.

You also have to think about the life cycle of code, if I make a small game in Unity and publish it, I'm probably not going to touch it after release.

ohgodplsno5• 4 years ago

when you need to write a functionality that is specific to your game, and it becomes too dependent on a certain engine or library, so you can't switch to another library or engine.

Games are not a thing you just change engines on. You don't paint with oil on a canvas then halfway through it go "you know what, I think aquarelle would be better" and swap out the canvas for aquarelle paper. The engine is a detail of your game and should never be core to it.

but then it doesn't leave enough room to add new cool things, like cool gameplay, a good multiplayer system, etc, when you want to ship on console who have limited memory.

You've not given a single look at Unreal, have you ? You have complete access to the entire source, it is modifiable and extendable to do absolutely anything you want, whether through blueprints or directly in C++. No cool multiplayer ? UE's default network replication is good enough to run on pretty much anything today. No cool gameplay ? ABZU (adventure puzzle exploring game), Assetto Corsa Competizione (highly accurate sim racing), Borderlands 3 (FPS Loot shooter), Dragon Ball FighterZ (fighting) as well as many experiments by users, including complex uses of portals and non euclidean geometry, etc.

It's not that the engine doesn't let you do cool things, it's that you're a software developer at heart that wants to find reasons to add new features to your engine, instead of making a game.

PUBG is a similar game, but its performances are quite bad

PUBG was made by a brand new developer who slapped together assets from the store and hacked away until it barely worked. The whole core of PUBG is rotten and terrible code. So, sure, with home made engine, you don't have this problem because someone like Playerunknown would literally not have had the skills to write the engine and the game would have never existed.

It's not an engine problem, it's a developer problem. Think about things for a minute when you do then and you'll have no troubles. It provides you with so many accurate and useful debugging tools that your home made engine will never have if you want to do a performance pass.

adbachman• 4 years ago

It's not that the engine doesn't let you do cool things, it's that you're a software developer at heart that wants to find reasons to add new features to your engine, instead of making a game.

This is so true. Yak shaving _feels_ incredibly productive. Look at everything I got done today!

The same can be true for rewrites, refactors, redesigns (product/graphic designers looove yak shaving), and prototypes.

I don't think that makes any of these things objectively bad. It just means there's a difference between wanting to have made a game and wanting to make games.

akyu• 4 years ago

Unreal and unity are monsters that allows beginners to have quick access to a high end rendering system, but then it doesn't leave enough room to add new cool things, like cool gameplay, a good multiplayer system, etc, when you want to ship on console who have limited memory.

I'm not following you here. These engines are exactly what enable you to add new cool things, and not focus on implementing lower level details.

booi• 4 years ago

Not to mention the huge library of games from B to AAA that use unreal/unity with things like cool gameplay, multiplayer and multi platform (PC and consoles).. this comment made no sense.

fb03• 4 years ago

Happy to see you here, Alice and the whole Bevy gang!

alice-i-cecile• 4 years ago

Finally blogging! I'm looking forward to writing and sharing more; it's been fun to hear everyone's perspectives.

Ericson2314• 4 years ago

Lol they are playing themselves a bit because stuff being "more than libraries glued together" is generally a bad thing!

hgs3• 4 years ago

GitHub is littered with failed projects taking this approach...

I would argue this is because most of the developers trying to make these game engines are hobbyists or students. Most professional software engineering these days is all about gluing packages together so anyone with modest work experience shouldn't have trouble with this.

seba_dos16• 4 years ago

And what's the definition of "failed" anyway? Lots of these projects are likely complete - served their purpose already and don't have to be touched anymore.

Animats3• 4 years ago

I'm painfully aware of this problem. I'm writing a metaverse viewer in Rust, to connect to the Second Life / Open Simulator servers and get better performance and look from that 20-year old technology. A metaverse viewer has the problems of both an MMO game client and a web browser. It works much like an MMO client, but it's more general. All the game logic and content come from the server. It faces most of the overload problems a web browser faces. Plus some of the hostile content problems.

I can use many library-type components from game ecosystems. But I can't use the big game engines, Unity and Unreal, because they assume a workflow that uses their own editors, their own networking, and their own formats. So I'm forced to glue libraries together. Some of which I have to write.[1] Most of it is little stuff. Too much little stuff.

That section about struggling to get open source library maintainers to fix things is spot on. I have Github issues open for six libraries. If I open the hood on one of those libraries, I could spend months getting deep enough to contribute. I can only do so much.

[1] https://github.com/John-Nagle/serde-llsd

seba_dos16• 4 years ago

If I open the hood on one of those libraries, I could spend months getting deep enough to contribute.

Really? I can understand it in case of missing features or stuff that needs deeper refactors, but in my experience digging into an external library to debug an issue with it usually ended up with a working patch at most a few days later (usually hours), which is comparable to finding workarounds but with much better outcome. Granted, a working patch doesn't necessarily mean a mergeable patch yet (though it often does for bug fixes), but with FLOSS dependencies you can simply ship a patched version before it gets accepted anyway.

Since my stack is fully FLOSS, I tend to look for fixes first before resorting to workarounds and this approach has served me pretty well so far. Even if I fail to come up with a fix in the end, I learn possible workarounds in the process.

Animats3• 4 years ago

Typical issues:

* winit - window.with_maximized ineffective on X11 platform

* rend3 - Minimize window causes unwrap panic on Windows / Wine. OK on Linux

Cross platform full screen is hard. Winit, WGPU, Rend3, and egui are all involved. Some things work on some platforms but not others. Order of operations matters where it shouldn't. (Must be visible before maximized on some platforms, but not all.) Right now I have enough workarounds to continue working. This area is brittle and easy to break. You need direct access to every supported platform (which, for winit, is Linux/X11, Linux/Wayland, MacOS on both x86 and ARM, Android, and WebAssembly) to work on this, and I don't have a full set of target machines. That comes up a lot with the libraries that handle cross-platform issues and present a portable API to the caller. Rust is good about cross-platform portability. It's impressive how well it works. But problems remain.)

* egui - Strange scrolling behavior - jitter up and down

Egui (a menu/widget overlay system) is under "heavy development". This problem is in the mechanism for scrolling text windows efficiently when most of the text is offscreen, a common choke point in GUI libraries. Need to chase this down. I built a standalone program to demo this, but haven't tried to fix it. This I probably do need to address myself.

* sys-locale - Returns "en_US" on Linux. Should return "en-US".

One of those little things that breaks other things. RFC 5646 says use "-", and the parser for locales assumes that. Easier to work around than fix, although I reported it. A fix will probably break something that's depending on the current wrong behavior. I don't know enough about the internationalization ecosystem to be confident changing this.

Game engines are more than libraries glued together

Related Stories