Making an RPG in Clojure (2010)

by arm • 10 years ago

191 39 10 years ago

Morgawr4• 10 years ago

Reading this article is like reading something I might have written myself. I, too, just like the author, have been toying around the idea of Clojure games for a long time. I've made small games (Ludum Dare entries mostly) and started (and subsequently killed) a lot of Clojure game engine projects.

I tried the threading approach using agents (which later evolved into agents inside atoms or was it atoms inside agents? It's been a long time) and came to the same conclusion as the author. I tried to move to a more centralized CES approach, which didn't work out.

It's interesting to see how some people can come down to the same path and achieve the same results (for the good or for the bad). I think my experience with Clojure has been overall positive as far as game development goes, although definitely not empty of woes.

Currently what I'm (very on and off) working on is a Visual Novel/Interactive storytelling engine on ClojureScript, at http://novelette.moe and Clojure is great for it. Feel free to trawl through my github profile (link in my profile) if you're curious about some of my (failed? abandoned?) Clojure gamedev projects (among other things).

priyankt• 10 years ago

Nice work with - http://morg.systems/ld31/ Not a lot of graphics & interaction, but the story telling was good.

I finished the game :-)

vanderZwan• 10 years ago

Nice, for some reason I have a soft spot for interactive fiction engines! How does it compare to Twine?

Morgawr4• 10 years ago

I had never heard of Twine before, I just had a quick superficial look at it and wow it looks really nice!

To be honest I don't know how it compares, my engine is closer to something like Ren'Py but I plan to provide more freedom to the writers thanks to the power of functional programming and the DSL-like implementation of Lisp. I haven't formalized on a proper syntax for the scripts yet (although a rough example can be seen on my demo game: https://github.com/Morgawr/ld31/blob/master/src/cljs/game/sc... ), but the idea is to have something like the script of an actual novel. The author can define acts and chapters and have actors enter the scene, interact, define the state of the scene (add/remove sprites, backgrounds, music) and define the transitions from chapter to chapter within the novel.

PaulHoule2• 10 years ago

The article also shows up that there is snake oil in the actors, stm and many other concepts to make concurrency easy and fun. Often you fall back on doing it all in 1 thread because you get better performance that way but then what is the poll of multicore?

Lmax disruptor is different from the others because it is based on engineering (what works on actual hardware) opposed to 'science' (how you imagine it ought to work)

meric• 10 years ago

> poll of multicore

Intended pun involving threads?

dr_zoidberg2• 10 years ago

I'd guess the keyboard (from a mobile device?) got "poll" instead of "pool". Or maybe a little typo from the use of writing one and not the other.

On the subject of pools of threads, a few days ago there was a post about the Naughty Dog Engine using fibers (inside threads) for cooperative multiprocessing. They made it so it'd have a cleaner API and be easier to use, but at the end of the talk they said it had also improved performance (because of less context switches).

Also having very little knowledge in games programming (a few experiments in QBasic 4.5 something like 20 years ago, when I was still a child :D ), the most I can get out of this is: multithreading/multiprocessing in games is hard.

vvanders2• 10 years ago

I remember seeing games doing actor-like things with message passing on a single thread way back in '02. You could "broadcast" a "damage" message and all the actors that were in the hit region would handle it in however they saw fit. Great way to force decoupling.

Concurrency is somewhat orthogonal(although a nice-to-have), it's more about not doing things that run counter to your hardware.

Over in the games space we don't have the advantage of scaling wide on multiple machines so a couple things that sound heretical are there for good reason.

Morgawr4• 10 years ago

You could "broadcast" a "damage" message and all the actors that were in the hit region would handle it in however they saw fit. Great way to force decoupling.

This is great for simple games but it gets hairy really quickly for larger games and it can easily become a mess. There are a few event/callback driven game engines out there and a lot of the most popular AAA game engines provide some kind of event subscription model (I think UE does it, at least, but I never really used it much).

The biggest problem is with figuring out who is in charge of what. Let's say I broadcast the event "player attacks enemy", then the enemy has to receive that event and respond with "enemy plays hurt animation" then "enemy plays dead animation" "enemy broadcast death" and then "player plays acquire XP animation" "player broadcasts level up" etc etc. The order can easily become a mess. And then "player attacks enemy" and "enemy evades attack" and "player plays missed animation" etc etc...

It really becomes very messy for big games, especially if you want to build an game engine entirely based on the broadcasting of events. I think Angel-Engine, now called Angel2D iirc, was mostly based on that and it worked fine for small scale stuff but easily become a mess.

It certainly is easy to understand the logic behind the game that way, it's great for small indie games or game competitions/prototypes, though.

_halgari• 10 years ago

No, it's just that things like actors and STM, and yes, even LMAX are completely overkill for many of the projects out there. Go read up on what LMAX was designed to solve, millions of messages a second. I've worked on several very large projects for huge corporations, where their message load was 1 thousand messages a second peak.

So yea, I don't use STM much, and I never use actors, I use queues quite a bit, but I benchmark them first and use the one that fits best with existing infrastructure and causes the least amount of friction with my client while still getting the job done. It has nothing to do with "actually working on hardware" or "science". Except perhaps that I apply the scientific method in choosing my tech, and that never ends up being LMAX for the jobs I work on.

PaulHoule2• 10 years ago

Actually lmax disruptor is great for all sorts of things, because 10 million messages per second allows fairly fine grained parallelism for data processing and also interactive apps like that app where you have back pressure support that keeps the app from being unresponsive. Real time work means not making commitments you can't keep so if it is clear the system cannot handle a request, drop the message early.

I mainly use the reactor framework, and I have a choice of lmax and a bunch of other queues and also the choice to schedule on the same thread with low overhead.

dkersten• 10 years ago

Well, I wouldn't call these techniques snake oil, but, just like anything, you can't just throw them at a problem and expect that they will just work. You have to take all of the usual concurrency issues into account (eg contention).

STM might make things simpler, but if many threads are contending for the same data, then they will need to be serialised and while STM might make this serialisation easier for the programmer, it won't automatically make it perform better[1].

For parallel threads to be faster than a single thread, you need the threads to be as independent from each other as possible. You can use actors, STM and such to do this, but its not a silver bullet that just works. You need to apply the correct technique given your situation and requirements.

I think where Clojure largely failed (and I say this as a big-time Clojure enthusiast) is in its promise of making concurrency easy through these magical primitives. Hell, since its release, Clojure has gained more concurrency primitives (futures, core.async, transducers) because the original ones (threads, STM, agents, atoms, vars) weren't enough.

As for a game engine, I would make sure that each entity wrote to its own independent data, so that writes never block each other. That is, they can all read global world state and they can read the previous state of the other entities, but each only ever writes its own data to its own atom (or whatever; I'd have to think about it more to figure out the most appropriate place to store the state). Then I can update all of the atoms in parallel. This is just off the top of my head, so it may need a little refinement. Agents provide a nice async update, but if you have many in flight, they still need to serialise and take turns, losing much or all of the parallelism.

The point is, I don't believe we'll have a magic "throw cores at it" concurrency technique any time soon, we will continue having to think about it and structure our code to keep the cores as busy as possible and synchronisation as low as possible.

LMAX disruptor is indeed a fantastic piece of engineering and it works so well because it eliminates the need for synchronisation (because the updates always happen in a single thread). The awesome "concurrency" from disruptor comes from the design of their ring buffer, which allows many threads to quickly write to it, and the single reader to quickly read, all without synchronisation. Basically, the processing thread can be kept busy without ever having to wait on writers. Fantastic!

My favourite means of parallelism, I think, is still a work-stealing task-based system like what Intel Threading Building Blocks provides, because its relatively easy to think about and deal with, quite flexible (allows firing off tasks for async things, "going wide" for nicely data-parallel things, allows pipelined processing for sequential steps over a collection of data).

But... there's no concurrency/parallelism silver bullet.

[1] STM can of course make sure the locks are as fine-grained as possible, but at the end of the day, if two things write to the same place, then they need to happen serially.

wtetzner2• 10 years ago

Well, based on the original videos Rich did to promote Clojure, I think I remember him claiming that STM was never meant for cases where you wanted parallelism for performance reasons.

STM was intended to make it less error prone to use threads in places where concurrency was the right fit for the problem.

MrBuddyCasino• 10 years ago

Yeah I remember the benchmarks, didn't look too good. People claimed that with hardware support for transactional memory things could be different. We have that now in the latest Intel CPUs (Skylake for sure), anyone got further information?

anamexis• 10 years ago

When is parallelism ever needed but for performance reasons? (Honest question, I'm quite a novice when it comes to concurrency)

andrewclunn• 10 years ago

Article is from 2010. Was there ever a follow up?

dr_zoidberg2• 10 years ago

That seems to be his last post on gaming/games: http://briancarper.net/tag/66.html

arm• 10 years ago

Unfortunately not, it seems like many factors in his life led to him burning out. He does have a new blog¹ now though.

――――――

¹ — http://carper.ca/

davexunit• 10 years ago

For those that are interested in what the current state of Lisp game development is, check out the latest installment of the Lisp Game Jam. The jam page includes lots of information about game development tools for many different Lisps, and the submissions include games written in Common Lisp, Scheme, Clojure, and Emacs Lisp.

https://itch.io/jam/spring-2016-lisp-game-jam

alexkehayias• 10 years ago

I've been working on a project to create 2D sprite based game engine in ClojureScript. My goal was to represent the game completely with data and let the engine figure out how to interpret and render it (declaratively). In the end I implemented an entity component system (as other comments have suggested) and used existing javascript rendering libraries to handle rendering (not writing a wrapper for webgl...). You can check it out here: https://github.com/alexkehayias/chocolatier

It's been a ton of fun bringing a functional approach to games using Clojure and I gave a talk about it at ClojureWest this year. https://www.youtube.com/watch?v=TW1ie0pIO_E hope this helps others looking to build games!

sdegutis• 10 years ago

This is great. I've also tried my hand at making video games in Clojure.

For me though, it was a weak attempt at a simple Bomberman clone that would be playable over the network, so that my kids could play it across our different family computers. It was even kid friendly, being about planting timed water balloons, and avoiding getting wet!

It seemed like Clojure would be ideal for this, because the JVM has excellent networking libs and GUI libs for simple 2d graphics.

But I severely underestimated how much time it would take though, so it never really even got to a playable single player state.

The main part I got stuck on was trying to settle on a data format, how to represent tiles and players and items and everything, in a form of immutable data that was easy to both analyze and transform.

The author's idea of mapping a giant 2d string to actual data is genius, although I suspect it wouldn't scale well for an RPG, and for my game it wouldn't have been very helpful since the terrain would be auto-generated (since needing to be somewhat randomized).

It's surprising to see the author here mention using Agents as the original primary building block of the game engine. I've never directly used Agents or found them helpful. To this day I really don't know what they're good for.

sbov2• 10 years ago

due to multiple threads trying to edit the world ref 50+ times per second.

Of course you get no benefit if you serialize the game through a single ref. Just because you have a fancy feature like agents doesn't mean you don't have to think about how your concurrent code is behaving.

I chose Clojure for the server side code of my MUD. It wasn't for the agents or STM, it was for immutability. Immutability gives me the ability to easily have a consistent snapshot of the world which I can use to perform some operation, such as saving, event propagation, or some AI calculations.

corecoder• 10 years ago

Which MUD?

lgas• 10 years ago

If you don't serialize the game through a single ref (or the equivalent) how do you get the consistent snapshot of the world to perform an operation on?

glhaynes• 10 years ago

Event loops are heavily used — basically process input, process physics, process NPCs, render, repeat. (Likely not in that order — I'm not much of a game developer.) This is what the author lands on in the "Success" section.

Speaking of event loops, I feel like they weren't discussed very much at all at my school, despite being at the core of not just most game engines, but most GUIs. I think if I'd embarked on writing a GUI system when I was fresh out of school, I'd probably have done it in a massively threaded way like the developer in the article started their game. An event loop would have somehow seemed too … primitive.

sbov2• 10 years ago

Architecture can heavily affect how serial your process is. By randomly throwing agents at a single ref you can easily turn parts of the process that can be done in parallel into a serial process.

Beyond that, throwing massive concurrency at an inherently serial process tends to give you worse performance vs just recognizing that it's a serial process and not trying to throw concurrency at the problem. This isn't just about fancy concurrency primitives like agents or STM, I've seen plenty of hobby games that ran terribly because they threw concurrency at the problem without analyzing it first, but ended up completely serial because of critical sections in the form of Java synchronization blocks.

As an example of concurrency that you would probably have in every single player game: game logic feeds into render logic, but render logic shouldn't be changing the state of the world. So you can be performing the game logic of game loop #N+1 while the render logic of game loop #N is running. Also, usually, when saving the state of the game all you need is a consistent snapshot at the time. Immutability makes both of these easy.

The rest depends upon the features and design of your game though. If you aren't going to actually try to analyze where you can benefit from concurrency, you're better off not just randomly throwing concurrency at the problem.

xiaoma• 10 years ago

This reminds me of Realm of Racket.

Elrac• 10 years ago

I love that the author takes us on a tour of all his blunders, so we can learn from them the way he did.

Far too many "making of" articles present you with a finely polished end product and pretend that its evolution was child's play. That leaves us readers disheartened and reluctant to try similar things.

resatori• 10 years ago

A few years ago I created a fully small RPG game in clojure: https://github.com/damn/cdq

Including procedurally generated maps, items and 3 small levels.

Making an RPG in Clojure (2010)

Related Stories