Rendering a game in real time with AI

https://blog.jeffschomay.com/rendering-a-game-in-real-time-with-ai

by jschomay • 11 months ago

114 109 11 months ago

blog.jeffschomay.com

The "real-time" version looks awful with constantly shifting colors, inconsistently sized objects, and changing interpretations of the underlying data, resulting in what I would consider an unplayable game vs the original ASCII rendering.

The "better" version renders at a whopping 4 seconds per frame (not frames per second) and still doesn't consistently represent the underlying data, with shifting interpretations of what each color / region represents.

faeyanpiraat• 11 months ago

Yeah, but I find this fascinating regardless.

This is getting into the direction of a kind of simulation where stuff is not determined by code but a kind of "real" physics.

roxolotl• 11 months ago

Why does using a language/vision model feel more “real” to you than using equations which directly describe our understandings of physics?

actuallyalys• 11 months ago

Yeah, as interesting as the concept is, the lack of frame to frame consistency is a real problem. It also seems like the computing requirements would be immense—the article mentions burning through $10 in seconds.

elpocko2• 11 months ago

You can do this at home on your own computer with a 40x0 consumer GPU at 1-2 fps. You have to choose a suitable diffusion model, there are models that provide sub-second generation of 1024x1024 images. The computing requirements and electricity costs are the same as when running a modern game.

harph• 11 months ago

It seems it's because OP is generating the whole screen every frame / every move. Of course that will give inconsistent results.

I wonder if this approach would work better:

1. generate the whole screen once

2. on update, create a mask for all changed elements of the underlying data

3. do an inpainting pass with this mask, with regional prompting to specify which parts have changed how

4. when moving the camera, do outpainting

This might not be possible with cloud based solutions, but I can see it being possible locally.

johnfn• 11 months ago

The "real-time" version looks awful, etc

Dang man it's just a guy showing off a neat thing he did for fun. This reaction seems excessive.

johnnyanmac11• 11 months ago

criticism is how we grow. And it seems we're more and more afraid for some reason to criticize when "AI" is involved.

ozmodiar• 11 months ago

I like the idea behind https://oasis-ai.org/ where you can actually try to take advantage of the 'dream logic' inconsistency of each frame being procedurally generated based on the last one. For example, instead of building a house, build the corner of a house, look at that, then look back up and check if it hallucinated the rest of your ephemeral house for you. Of course that uses AI as the entire gameplay loop and not just a graphics filter. It's also... not great, but an interesting concept that I could see producing a fun dream logic game in the future.

jschomay3• 11 months ago

OP here. Thanks for the feedback. I agree that frame to frame consistency is quite bad currently. I did address that in the post, hinting at some of the techniques others have mentioned here, like in/out-painting and masking previous frames. For me, the exciting parts of this experiment was finding the opportunities and limits of realtime generation, and exploring ways of grounding generated content in a solid yet player controlled world layer.

g105b• 11 months ago

I've been trying to achieve the opposite of this project: render scenes in ASCII/ANSI in the style of old BBS terminal games. I've had terrible success so far. All the AI models I've tried only understand the concept of "pixel art" and not ASCII/ANSI graphics such as what can be seen on https://www.bbsing.com/ , https://16colo.rs , or on Reddit's r/ANSIart/ .

If anyone has any tips for how I could achieve this, I would love to hear your ideas.

elpocko2• 11 months ago

Do you mean you want to use AI to generate new scenes in ANSI-art style, or do you mean you want to use AI to render pre-existing scenes as ANSI art?

ticulatedspline• 11 months ago

like this https://asciiart.club/ ? you don't really need AI for that.

if you have to use AI diffusion models are a poor choice for the target. might be easier to train an LLM to output actual text art.

pjob• 11 months ago

I suspect you'll have much better luck directly implementing the style you want rather than trying to get a (current gen) AI to reliably generate ANSI art. This video is a nice overview of implementing a variation of the style in a post-processing shader:

https://www.youtube.com/watch?v=gg40RWiaHRY

cbm-vic-20• 11 months ago

...where "ASCII" means an image made up of a grid of elements from a limited set of glyphs.

tantalor• 11 months ago

And those glyphs are not ASCII

This is ASCII: https://commons.wikimedia.org/wiki/File:ASCII-Table-wide.svg

mason_mpls3• 11 months ago

If we’re talking about dwarf fortress it uses an old IBM charset, assuming this is some branch off that

panki27• 11 months ago

I'm pretty sure the generation could easily run locally on a low-to-mid tier graphics card.

While it might take a bit longer to generate, you're still saving network and authentication latency.

echelon2• 11 months ago

This was built in September 2022, and it's still pretty mind-blowing today:

https://madebyoll.in/posts/game_emulation_via_dnn/demo/

https://madebyoll.in/posts/game_emulation_via_dnn/

Hook world state up to a server and you have multiplayer.

2025 update:

https://madebyoll.in/posts/world_emulation_via_dnn/

https://madebyoll.in/posts/world_emulation_via_dnn/demo/

jschomay3• 11 months ago

That's really cool, thanks for sharing!

stego-tech• 11 months ago

…but then you just have a graphics card, built to render graphics, that you could tap instead through traditional tooling that’s already widely known and which produces consistent output via local assets.

While the results of the experiment here are interesting from an academic standpoint, it’s the same issue as remote game streaming: the amount of time you have to process input from the player, render visuals and sound, and transmit it back to the player precludes remote rendering for all but the most latency-insensitive games and framerates. It’s publishers and IP owners trying to solve the problem of ownership (in that they don’t want anyone to own anything, ever) rather than tackling any actually important issues (such as inefficient rendering pipelines, improving asset compression and delivery methods, improving the sandboxing of game code, etc).

Trying to make AI render real-time visuals is the wrongest use of the technology.

tliltocatl• 11 months ago

Nah, vibe coding is the wrongest use of technology, this is way to go. Why? Because good rendering isn't the necessarily most physically accurate one. You might actually want non-realistic rendering, and (depending on the specific style) it might be hard to impossible to get the right one on traditional pipeline. E. g. take "cartoonish" look - toon shading is, frankly said, total crap, because the artists rely on explicitly non-physical geometry to provide visual clues. This is definitely the future. Render on a normal pipeline (maybe with no lighting model at all), then put it through a style transfer network.

johnisgood4• 11 months ago

I do not see how it replaces or substitutes network and authentication latency, especially in terms of a single player game in which neither is necessary.

steveruizok2• 11 months ago

We did a similar thing at tldraw with Draw Fast (https://drawfast.tldraw.com/) and it was very fun. Inspired a few knock offs too. We had to shut it down because it was getting popular on Russian Reddit. A related project Lens (https://lens.tldraw.com) also used the same technique, but in a collaborative drawing app.

At the peak, when we were streaming back video from Fal and getting <100ms of lag, the setup produced one of the most original creative experiences I’d ever had. I wish these sorts of ultra-fast image generators received more attention and research because they do open up some crazy UX.

echelon2• 11 months ago

LCM is what Krea used to gain massive momentum and raise their first $30M.

The tactile reaction to playing with this tech is that it feels utterly sci-fi. It's so freaking cool. Watching videos does not do it justice.

Not enough companies or teams are trying this stuff. This is really cool tech, and I doubt we've seen the peak of what real time rendering can do.

The rendering artifacts and quality make the utility for production use cases a little questionable, but it can certainly do "art therapy" and dreaming / ideation.

dvrp• 11 months ago

hey thanks for the kind words (krea cofounder here). i agree with that! everyone’s focused on leaderboards and evals but not enough on higher order repercussions imo

on a related (but shameless plug) note, did you check our real time video rendering release?

https://x.com/krea_ai/status/1961074072487620635

hiatus• 11 months ago

Is there any chance you'd open up the source for those projects so others can play with them?

Topfi• 11 months ago

They've already shared it under a none-commercial use license: https://github.com/tldraw/draw-fast

The TLDraw team, from what I have seen, is really open in sharing their experiments under their own license. Not FOSS strictly, but I feel their licensing approach and decision is fair considering how they fund development, know that it's a contentious topic though for some.

jschomay3• 11 months ago

OP here, I remember both of the draw fast and lens demos! I'm pretty sure those were in the back of my subconscious, inspiring me to explore my take on real time rendering. Thanks for sharing your similar experience. I agree, this was a lot of fun to work on, and like one of the other commenters pointed out, experiencing it viscerally is a whole new kind of feeling, even with the consistency issues. I'd also like to see more experiments on what new kinds of UX could be possible with this tech.

_1tem• 11 months ago

This is a pre-cursor to a dystopian future where reality will be a game generated in realtime at 60 FPS and streamed to your brain over Neuralink.

lm28469• 11 months ago

Some people on HN will surely cheer for it, it's the peak of efficiency, you don't even have to move anymore!

johnnyanmac11• 11 months ago

Yup, only the 2nd or 3rd most common trope of cyberpunk dystopias. We clearly just want the world to end at this point.

mason_mpls3• 11 months ago

I think that’ll be one of the few good parts of it imho

jebarker• 11 months ago

In some ways that’s a lot like how consciousness works isn’t it?

coolKid7214• 11 months ago

I do not get the point of this at all, why not just generate game assets and run them in an engine? With this format there would be no regularity that the thing you saw before will look the same (and that is not a fixable problem).

Actually figuring out and improving AI approaches for generating consistent and decent quality game assets is actually something that will be useful, this I have no idea the point of past a tech demo (and for some reason all the "ai game" people do this approach).

abbycurtis33• 11 months ago

The tech will improve to far exceed the capabilities of a game engine. Real time improvisation and infinite choices, scope, etc.

It makes no sense when people say AI can't do this or that. It will do it next week.

lukan3• 11 months ago

"It makes no sense when people say AI can't do this or that. It will do it next week."

So full self driving vecicles will be finally ready next week then? Great to hear, though to be honest, I remain sceptical.

1272• 11 months ago

An interactive feedback loop that handles various edge cases of AI, rendering it, asset loading and display, keeping track of global data, user input, etc. -- is still a game engine.

GPerson• 11 months ago

I’m looking forward to the day when magical thinking such as this gets grounded again. That is when the real work will start anew.

corysama4• 11 months ago

Having spent 20 years making game engines and the past 4 years playing with AI image gen, I believe you are right.

There have been musings for a while now that 3D rendering is going to switch from “lay down the scene’s albedo & specular parameters then do the lighting pass” to “lay down the scene’s latent parameters and then do the diffusion pass”.

Recently, the advances in “real time AI world models” have been coming ridiculously fast.

Put these together and it’s no stretch at all imagining a game built by having artists go nuts doing whatever they want with whatever Maya can handle as long as they also make proxy geometry of trivial complexity that can be conceptually associated with the final renders. Train the AI on the association. Render the proxy geometry the old fashioned way. AI that up in real time to the associated Maya-final-render approximation.

It’s not going to happen this week. But, in 5 years? Somebody’s gonna pull it off.

fzeroracer• 11 months ago

Why would I want any of that? Games are interesting because of the deliberate choices and limitations made by a developer. When you have a game that tries to do everything, you have a game that actually does nothing.

johnnyanmac11• 11 months ago

I'm a game engine programmer and part of my job is to figure how how to "do it next week". I don't see it happening with the we we use LLM's right now. We don't have time to push data to a cloud and come back and render a scene, we have 16ms per frame on most projects (maybe 33 if you are doing something cinematic).

You're gonna have to break the light barrier before thinking about how to do real time Ai rendering at this rate.

dagi3d• 11 months ago

hack, learn and have fun, that's it.

johnnyanmac11• 11 months ago

What are we learning here, and are we still bound by ethics?

sho_hn2• 11 months ago

I do not get the point of this at all

Dunno, this seems like an avenue definitely worth exploring.

Plenty of game applications today already have a render path of input -> pass through AI model -> final image. That's what the AI-based scaling and frame interpolation features like DLSS and FSR are.

In those cases, you have a very high-fidelity input doing most of the heavy lifting, and the AI pass filling in gaps.

Experiments like the OP's are about moving that boundary and "prompting" with a lower-fidelity input and having the model do more.

Depending on how well you can tune and steer the model, and where you place the boundary line, this might well be a compelling and efficient compute path for some applications, especially as HW acceleration for model workloads improves.

No doubt we will see games do variations of this theme, just like games have thoroughly explored other technology to generate assets from lower-fidelity seeds, e.g. classical proc gen. This is all super in the wheelhouse of game development.

Some kind of AI-first demoscene would be a pretty cool thing too. What's a trained model if not another fancy compressor?

troupo• 11 months ago

*Edit: misread the post I replied to, disregard comment contents*

Plenty of game applications today already have a render path of input -> pass through AI model -> final image.

Where "plenty" equals zero?

In those cases, you have a very high-fidelity input doing most of the heavy lifting, and the AI pass filling in gaps.

That is, in these cases you already have high-fidelity input in the form of an actual game, and "AI" contributions to the output are dubious at best.

Do you really believe it's DLSS that's doing all the heavy lifting in a game like Expedition 33 or Cyberpunk 2077?

coolKid7214• 11 months ago

Compared to just coming up with and having solid systems for generating game assets? Actually having decent quality style consistent 3d models, texture work, animation, sound effects, etc (especially if it were built into say a game engine or something) would actually revolutionize indie game dev. Games are fundamentally artistic works so yes anything decent will actually require tailoring and crafting, AI set up to serve those people actually makes sense and is totally technically feasible are way easier problems to solve.

And no if you heavily visually modify something with AI models to the extent it significantly alters the appearance it simply has no way of being consistent unless you include the previously generated thing somehow which then has the huge problem of how do you maintain that over an 80 hour game? How do you inform the AI what visual elements (say text, interactive elements) are significant and can be modified and which can't? (You can't)

Actually using AI to generate assets, having a person go in to make sure they look good together and make sure they match then just saving them as textures so they function like normal game assets makes 10000x more sense then generating a user image then trying to extract "hey what did the wrapper of this candy bar look like" from one ai generated image and figuring out how to make sure that is consistent across that type of candy bar in the world and maintains that consistency throughout the entire game, instead of just you know, generating a texture of a candybar?

turnsout• 11 months ago

It's an interesting tech demo—I think one interesting use case for AI rendering is changing the style on the fly. For example, a certain power-up could change the look to a hyper-saturated comic book style. Definitely achievable with traditional methods, but because AI is prompt-based, you could combine or extend styles dynamically.

1272• 11 months ago

Visually speaking, there's always visual issues in tying disparate assets together in a seamless fashion. I can see how AI could be easily used to "hide the seams" so to speak. I think a hybrid approach would be an improvement definitely.

shortrounddev2• 11 months ago

AI can't even hide its own seams. The seams are kind of the defining characteristic of AI.

Workaccount22• 11 months ago

(and that is not a fixable problem)

Genie 3 is incredible (relative to previous models) in this regard. Not a solve, but it is doing what it is not supposed to be able to do.[1]

[1]https://deepmind.google/discover/blog/genie-3-a-new-frontier...

corysama4• 11 months ago

Here’s a recent demo made by researchers at Nvidia trying to render the most detailed, realistic scene possible using AI tech to a small degree -mostly as asset compression, but also in material modeling beyond what direct approaches can currently handle in real time: https://m.youtube.com/watch?v=0_eGq38V1hk

Here’s a video from a rando on Reddit conveniently posted today after playing around for an afternoon: https://www.reddit.com/r/aivideo/comments/1n1piz4/enjoy_nano...

The Nvidia team carefully selected a scene to be feasible for their tech. The rando happened to select a similar-ish scene that is largely well suited for game tech.

Which scene looks more realistic? The rando’s by far. And, adding stylization is some AIs are very well established as being excellent at.

Yes, there are still lots of issues to overcome. But, I find the negativity in so many comments in here to be highly imbalanced.

coolKid7214• 11 months ago

that rando's one looks like fucking garbage like a bunch of shitty B roll footage from an AI generated advertisement for some kind of pharmaceutical, the space and layout of the world clearly is constantly shifting and makes no sense. just generate a fucking map we already know how to do this, tracking geometry is an EASY problem for computers, don't force some AI shit to try to do it for some reason.

sensanaty2• 11 months ago

There's something so unsettling about that 2nd link you posted, and I don't mean from an "AI is impressive" POV (I think it looks like absolute garbage but will probably continue improving bit by bit). There's something even in the still frames which scratches my brain in a very unpleasant way, it's hard to describe the sensation. The closest thing is "disgust", maybe it's a uncanny valley type of effect.

Also, how are these 2 comparable at all? Obviously the video looks more "realistic", the first one is obviously a game demo of some kind and is stylized, whereas the latter looks like a terrible travel agency ad.

elestor• 11 months ago

I do not get the point of this at all

I think the point of this is just because it's cool. So, as you said, it only serves as a tech demo, but why not? Many things have no point. It's unreasonable, but it's cool.

mason_mpls3• 11 months ago

Now dwarf fortress can eat your CPU, Memory, and GPU. Exciting news.

beacon294• 11 months ago

I thought so too, but looks like the OP wrote their own game that just has a similar ascii view.

However, no reason this can't translate to df!

bob1029• 11 months ago

The power consumption of modern gaming is getting a bit out of hand. This AI stuff is taking it to the next level.

Ray tracing and other forms of real time global illumination are extremely resource intensive approaches to lighting a scene. Every client machine has to figure out how to light everything every single frame. Contrast this with baked global illumination where the cost is incurred exactly once and is amortized across potentially millions of machines.

We need more things like baked GI in gaming. This class of techniques makes the development iterations slower and more deliberate, but it also produces a far more efficient and refined product experience. I'd be very interested in quantifying the carbon impact of realtime vs baked lighting in gaming. It is likely a non-trivial figure at scale. Also bear in mind that baked GI is why games like the batman series still look so good in 2025, even when running on period hardware. You cannot replace your art team by consuming more electricity.

nh23423fefe• 11 months ago

It doesn't seem like moralizing about resources matters for velocity or capability.

You cannot replace your art team by consuming more electricity

This isn't true.

johnnyanmac11• 11 months ago

It doesn't seem like moralizing about resources matters for velocity or capability.

Not until the Hype bubble pops. Then suddenly it'll be all that matters when you no longer have nigh limitless resources.

This isn't true.

This project is interesting, but it is not making a great arguemnt for this.

Martin_Silenus• 11 months ago

This kind of lazy lamer behaviour is so unlike Vulkan and hackers...

Wait a minute. Where am I?

johnnyanmac11• 11 months ago

Vibe News, of course!

VagabundoP• 11 months ago

How far are we from Ender's Game?

devinprater• 11 months ago

Cool, if the game is promptable, one may even be able to make the game accessible.

doawoo2• 11 months ago

[flagged]

mediaman• 11 months ago

It's interesting to see software engineers realize that AI can be useful in the hands of competent engineers, but that LLMs tend to produce a mess in the hands of those with little software engineering knowledge. Then they're confused why generative AI asset creation doesn't look that good in the hands of people who have no art training.

I know someone creating a game, and she is using AI in asset creation. But she's also a highly skilled environment technical artist from the industry, and so the use of AI looks totally different than all these demos from non-artists: different types of narrow AI get injected into different little parts of the asset creation pipeline, and the result is a mix of traditional tools (Substance Designer + Painter, Blender, Maya) with AI support in moodboarding, geometry creation and some parts of texture creation. The result is a 2-5x speedup, but instead of looking like slop it looks like a stylistically distinctive, cohesive world with consistent art direction.

The common pattern is that people think AI will automate "other people," because they see its shortcomings in their own field. But because they don't understand the technical skill required in other fields, they assume AI will just "do it." Instead, it seems like AI can be a force multiplier for technically skilled people, but that it begins showing its weakness when asked to take over entire pipelines normally produced by technically skilled people, whether they be engineers or artists.

doawoo2• 11 months ago

Really interesting perspective honestly, I legitimately have no issue with machine learning being used as a tool, like all technology it has benefits!

It’s when it’s used as a replacement for creativity that really gets to me.

sovietmudkipz• 11 months ago

Sci-fi readers who’ve read Ender’s Game will recognize this style of software as similar in concept to the Mind Game Ender Wiggins plays. In the book, the Mind Game renders a game world based on the subject’s mind (conscious, subconscious) in a mechanically similar way to how dreams work for us IRL.

I’m excited for AI rendered games.

theknarf• 11 months ago

This is like playing a boardgame while exclusively looking at the board through a snapchat filter. Who wants this?

gavmor• 11 months ago

This is awesome! I can definitely see it delivering value, especially over with procedurally generated terrain with 100s or 1000s of different terrain types and combinations—especially if user-defined properties get involved, ie what artists can't predict or prepare for, producing materials via something like https://infinite-craft.gg/

babush• 11 months ago

Beautiful

Rendering a game in real time with AI

Related Stories