Porting SBCL to the Nintendo Switch

https://reader.tymoon.eu/article/437

by todsacerdoti • 2 years ago

415 81 2 years ago

reader.tymoon.eu

colingw• 2 years ago

I have being using Trial[1] for the past few weeks to test out game development in Common Lisp, and have been having a great time. Being able to alter (almost) all aspects of your game while it's running is a blessing.

I hope this port succeeds.

[1] https://github.com/Shirakumo/trial

sinker• 2 years ago

Lisp languages seem well-suited for building games. The ability to evaluate code interactively without recompilation is a huge deal for feature building, incremental development, and bug-fixing. Retaining application state between code changes seems like it would be incredibly useful. Common Lisp also appears to be a much faster language than I would have blindly assumed.

The main downside for me (in general, not just for game programming) is the clunkiness in using data structures - maps especially. But the tradeoff seems worth it.

pjmlp4• 2 years ago

There are 1980's papers about Lisp compilers competing with Fortran compilers, unfortunately with the AI Winter, and the high costs of such systems, people lost sight of it.

lispm• 2 years ago

One of the downsides is that implementations like SBCL have a deep integration and need things like a well performing GC implementation - to get this running on specialized game hardware is challenging. The article describes that. Getting over the hurdle of the low-level integration is difficult. The reward comes, when one gets to the point, where the rapid incremental development cycles of Common Lisp, even with connected devices, kicks in.

For the old historic Naughty Dog use case, it was a development system written in Common Lisp on an SGI and a C++ runtime with low-level Scheme code on the Playstation.

Common Lisp also appears to be a much faster language than I would have blindly assumed.

There are two modes:

1) fast optimized code which allows for some low-level stuff to stay with Common Lisp

2) unoptimized, but natively compiled code, which enables safe (-> the runtime does not crash) interactive and incremental development -> this mode is where much of the software can run nowadays and which is still "fast enough" for many use cases

koito17• 2 years ago

The ability to evaluate code interactively without recompilation

SBCL and other implementations compile code to machine code then execute it. That is to say, when a form is submitted to the REPL, the form is not interpreted, but first compiled then executed. The reason execution finishes quickly is because compilation finishes quickly.

There are some implementations, like CCL, with a special interpreter mode exclusively for REPL-usage.[1] However, at least SBCL and ECL will compile code, not interpret.

[1] https://github.com/Clozure/ccl/blob/v1.13/level-1/l1-readloo...

arktos_• 2 years ago

There are some libraries that make maps and the like usable with a cleaner syntax. You too could make some macros of your own for the same purpose, if syntax is the concern

Jach5• 2 years ago

If that's your main downside, that's pretty good, since clunkiness is in many ways fixable. Personally with standard CL I like to use property lists with keywords, so a "map literal" is just (list :a 3 :b 'other). It's fine when the map is small. The getter is just getf, setting is the usual setf around the getter. There's a cute way to loop by #'cddr for a key-and-value loop, though Alexandria (a very common utility library) has some useful utils for looping/removing/converting plists as well.

If typing out "(list ...)" is annoying, it's a few lines of code to let you type {:a 3 :b 4} instead, like Clojure. And the result of that can be a plist, or a hash table, or again like Clojure one of the handful of immutable map structures available. You can also easily make the native hash tables print themselves out with the curly bracket syntax.

(On the speed front, you might be amused by https://renato.athaydes.com/posts/how-to-write-slow-rust-cod... But separately, when you want to speed up Lisp (with SBCL) even more than default, it's rather fun to be able to run disassemble on your function and see what it's doing at the assembly level, and turn up optimization hints and have the compiler start telling you (even on the individual function level) about where it has to use e.g. generic addition instead of a faster assembly instruction because it can't prove type info and you'll have to tell it/fix your code. It can tell you about dead code it removed. You can define stack-allocation if needed. Simple benchmarking that also includes processor cycles and memory allocated is available immediately with the built-in time macro...)

spsesk117• 2 years ago

Thanks to the author for the fascinating and detailed write up. It feels like a lot of the time this level of detail around the specifics of 'blessed' (not homebrew) console porting are only revealed years after the end of the consoles lifetime.

As an aside, reading about this kind of deeply interesting work always makes me envious when I think about the rote software I spend all day writing :)

cschep• 2 years ago

well said!

as I was just sitting down to another day of ruby on rails (that I am grateful for!) I was thinking.. I wonder what hobby/open source projects could use some of my attention later..

.. what projects my attention could use later .. :D

whizzter• 2 years ago

At least back when I was working these "blessed" tools were usually a tad hacked together, modern homebrew toolchains for many older platforms are better except for debugging support (since the devkits for the machines usually had better hooks available but also avoiding the entire GDB focus).

Having been in both worlds, i'm not entirely sure there's that much to be envious of.

fouric• 2 years ago

This is super neat - SBCL is an awesome language implementation, and I've always wanted to do CL development for a "real" game console.

I'm also surprised (in a good way) that Shinmera is working on this - I've seen him a few times before on #lispgames and in the Lisp Discord, and I didn't know that he was into this kind of low-level development. I've looked at the guts of SBCL briefly and was frightened away, so kudos to him.

I wonder if SBCL (+ threading/SDL2) works on the Raspberry Pi now...

f1shy2• 2 years ago

- Is it not her?

anisoco• 2 years ago

It is

https://reader.tymoon.eu/article/436

Shinmera10• 2 years ago

I'm not doing the SBCL parts, that's all Charles' work that I hired him for. My work is the portability bits that Trial relies on to do whatever and the general build architecture for this, along with the initial runtime stubbing.

And, as mentioned, *her :)

cellularmitosis• 2 years ago

Holy cow, Kandria looks amazing. Is it also developed using Trial? https://www.youtube.com/watch?v=usc0Znm-gbA

ducktective• 2 years ago

I wish the likes of Nintendo and Sony themselves finance such efforts. I mean it's one another way to create games (IP) for your console, what possibly could be the downside of starting something similar to Github Accelerator for your platform?

jsheard2• 2 years ago

Because it's well established that game developers can and will jump through whatever hoops the platform holder demands at their own expense, they don't have the leverage to be picky about the technical details when deciding which platforms to ship on. Nintendo doesn't need to create new incentives to release on the Switch when they already have the biggest incentive of all: 140+ million units sold, and a high attach rate.

At least there isn't as much hoop jumping as there used to be, since the systems have all converged on using commodity CPU and GPU architectures with at most minor embellishments.

Arelius3• 2 years ago

Yeah, also whatever they would build, the other platform vendors won't choose the same thing, and it wont be the exact variant of lisp or w/e that even the few nice developers would want.

I wish vendors would be just more supportive of different llvm tool chains. Rust isn't even well supported.

pjmlp4• 2 years ago

They did in the past and the only thing people did was to port MAME and other emulators, or copies from old 8 and 16 bit days.

That is why we cannot have nice things.

Retr0id4• 2 years ago

The answer to that is that while I would desperately like to share it all publicly, the NDA prevents us from doing so.

I'm curious what the rationale here was for using the official SDK, rather than the unencumbered "homebrew" ones[0].

As a complete guess, maybe Nintendo doesn't let you officially publish games built using 3rd party SDKs?

[0] https://switchbrew.org/wiki/Setting_up_Development_Environme...

Shinmera10• 2 years ago

You cannot publish games with homebrew, it has to use the official SDK. Besides that, almost nobody has a jailbroken Switch, so it would make it extremely hard to play any games on anything but an emulator.

Retr0id4• 2 years ago

You cannot publish games with homebrew, it has to use the official SDK

This doesn't surprise me much, but does Nintendo state this explicitly anywhere public?

If Nintendo chose to sign an application developed using a 3rd party toolchain, there's no technical reason why it couldn't run on retail consoles.

favorited• 2 years ago

almost nobody has a jailbroken Switch

This isn't really my scene so I don't know the details, but I remember reading that the first 10+ million Switches produced have an unpatchable bootloader exploit. I'm sure you're correct that almost nobody actually has a hacked console, but my understanding is that they're readily available for people who want one.

ArneBab2• 2 years ago

You don’t let your kids jailbreak their Switch. Because it’s a damn online system, so any leaked info and Nintendo can brick the Switch. And their game states are far too valuable for the kids for that.

noobermin• 2 years ago

This is what I come to HN for. Kudos to OP and their colleague. I know it's impossible but what a blessing it would be if Nintendo could be a little more open about their system.

whitten• 2 years ago

I probably missed it, but since the switch doesn’t have a keyboard how do you handle text input ?

jsheard2• 2 years ago

The Switch can support a USB keyboard, which would be the nice way to do it. There's already a fair number of officially published games with keyboard and/or mouse support, including a couple of programming ones. It has an on-screen keyboard too of course but you wouldn't want to rely on that more than absolutely necessary.

Arelius3• 2 years ago

Over the network most likely?

Shinmera10• 2 years ago

It does have a HID keyboard API, but typically you're expected to present an on-screen keyboard.

not_your_vase2• 2 years ago

Somewhat offtopic, just flashed through my mind: you know what would be amazing and absolutely useless at the same time?

Porting Yuzu to Nintendo Switch

laidoffamazon• 2 years ago

This has been done, and it’s been possible to run Switch on Switch for several months [0] (about 39 minutes into the video).

[0] https://youtu.be/H1gveQUBIKk

not_your_vase2• 2 years ago

Shoot, it was a suspiciously genius idea* for a Friday 13th.

*: Compared to my usual ideas

jfim• 2 years ago

Since I had to look it up, Yuzu is a Nintendo switch emulator.

sleepycatgirl• 2 years ago

Her work is just amazing. And makes me incredibly happy, as I like to write some CL here and there.

Shinmera10• 2 years ago

Aw, thank you. Happy hacking!

trenchgun• 2 years ago

Does anybody know what is the status of Trial on Mac OS? Specifically Apple Silicon

Shinmera10• 2 years ago

I don't have a mac, let alone a silicon one, so I can't test on it (I also have no patience for Apple's BS). However, it should work. At least SBCL itself runs, and Trial is mostly portable code, so it should, too, modulo some regressions.

nanna• 2 years ago

I've just bought Kandria. I'm not much of a game player so I probably won't get much play out of it, but Shinmera is clearly pushing the bounds of the Lisp world, and that's something to support.

hoten• 2 years ago

b/c it isn't described anywhere...

SBCL - "Steel Bank Common Lisp"

Steel Bank Common Lisp (SBCL) is a high performance Common Lisp compiler. It is open source / free software, with a permissive license. In addition to the compiler and runtime system for ANSI Common Lisp, it provides an interactive environment including a debugger, a statistical profiler, a code coverage tool, and many other extensions.

https://www.sbcl.org/

varjag• 2 years ago

"Steel Bank" in turn is a homage to "Carnegie Mellon" of CMUCL.

ArneBab2• 2 years ago

According to the Benchmarks Game, SBCL is roughly as fast as Node.

https://benchmarksgame-team.pages.debian.net/benchmarksgame/...

tmtvl• 2 years ago

See also https://github.com/attractivechaos/plb2 ...where I provided the SBCL solutions, so there's probably still a significant chunk of performance to be squeezed out.

drmeister• 2 years ago

Awe-inspiring work Shinmera and Charles.

dasyatidprime• 2 years ago

How's CL's GC performance for games nowadays? I've been slightly eyeing the upcoming Autumn Lisp Game Jam myself, but last I checked all the major libre CL impls, including SBCL, still used a full stop-the-world collector, which feels like a recipe for latency spikes. I saw flashes of stuff on sbcl-devel about someone working on a lower-latency one, but I don't know whether it got anywhere.

vindarel• 2 years ago

see this detailed report: https://raw.githubusercontent.com/Shinmera/talks/master/els2...

Overall we have needed to do surprisingly little actual performance analysis and optimisation work to make Kandria run well.

superdisk• 2 years ago

Great article. One question I had, not to diminish this hard work, is why not use a different implementation like ECL which is pretty portable already and can compile to static C code which can just be compiled traditionally for the target? I've been doing that for a Wasm + SDL2 game in Lisp and it's been (relatively) straightforward. I suppose performance might be a issue.

Shinmera10• 2 years ago

Because, as you guessed, no implementation other than SBCL comes close to the performance needed.

aktau• 2 years ago

Related: https://opengoal.dev.

Context: Naughty Dog used a custom Lisp-alike (GOAL) to build the Jak & Daxter series on PS2. They left enough debugging information in that it was possible to reverse engineer. The OpenGOAL project has done so, and these games can now be run on all platforms that their GOAL compiler gets ported to (x86 for now AFAIK). Would be cool to port this to the Switch.

terrycody• 2 years ago

How is this thing useful? Does it mean you can use the Trial to develop games completely free hassle on NS after its done, and you don't need a popular engine like Unity, Godot, Unreal?

rootnod3• 2 years ago

Absolutely kudos to that effort. I love hacking around with CL and some Schemes, and an effort like this is just amazing.

snthpy• 2 years ago

OT but is there an easy way to build a game with Godot or Unity and deploy it to the Switch? I'd love this for my son.

Porting SBCL to the Nintendo Switch

Related Stories