Dynamic music and sound techniques for video games

https://blog.gingerbeardman.com/2023/12/09/dynamic-music-and-sound-techniques-for-video-games/

by ecliptik • 3 years ago

101 36 3 years ago

blog.gingerbeardman.com

Razengan3• 3 years ago

Nothing will beat IMUSE from Monkey Island 2. Maybe because of the way MIDI worked versus the "pre-rendered" audio used today?

https://www.youtube.com/watch?v=AjtxK_WT784

msephton4• 3 years ago

I recently listened to a 2015 interview with Ron Gilbert where he discusses this, it's here: https://youtu.be/cktmhqXMsGI?t=3638 and summarised in this reddit comment: https://www.reddit.com/r/gamedev/comments/4bwnzm/comment/d1d...

Razengan3• 3 years ago

Maybe real-time AI generated music, at least for transitions, can eventually do the same thing iMUSE did:

https://www.youtube.com/watch?v=Lxxu1dVtg5g

crtified• 3 years ago

As a fellow Monkey Island 2 and iMuse nostalgiac, I also really enjoyed the user-controllable musical segue effect used in the recent game Sea of Stars.

https://www.youtube.com/watch?v=4dOG4DECEGk

(technically speaking I assume that is nothing like the MIDI-based iMuse, but rather, something like a cross-fade between 2 synchronised versions of the same song, but still! - it sounds neat, and makes me feel a little like iMuse did back in the 90s)

Musical invention and interaction is a wonderful aspect of video gaming.

thom• 3 years ago

Yeah there was a big push towards this in the 90s, I seem to remember the Wing Commander games and Ultima Underworld trying it. It's a similar tragedy to voiceovers in games - RPGs had increasingly interesting dialogue systems, Wizardry and Ultima would let you talk to NPCs using keywords or whole sentences. Voice acting killed that entirely. One can only hope that the era of LLMs brings back some of this interactivity.

Razengan3• 3 years ago

One can only hope that the era of LLMs brings back some of this interactivity.

The Wheel turns, and things Old become New again.

temp0826• 3 years ago

My first thought was Untitled Goose Game! So cheeky and fun how the music works in it

crazysim• 3 years ago

Fun fact: they built that in after their first trailer was made.

The "dynamic" music had apparently set some unexpected expectations.

rzzzt• 3 years ago

Mine was Soldier of Fortune. It blends smoothly between high alert and sneak-around segments, eg. tracks 2 and 3 on the first level: https://www.youtube.com/watch?v=ZI_14lp1U2w&list=PLALRZSnKU7...

koen_hendriks• 3 years ago

I was thinking about SSX tricky and 3. How the music loops and/or fades in for example big air time or landing a trick.

There were certain layers of the playing track that would fade away when in air such as vocals. I always thought that this was done really great in these games.

devmunchies• 3 years ago

I mostly use royalty free music of Japanese origin

Like what? I’d like to hear what royalty free Japanese music sounds like.

msephton4• 3 years ago

Here's the music that I used in my most recent game (which I call: Fore! Track): https://twitter.com/gingerbeardman/status/173255553386375169...

This track is by an amazing, long-standing musician that goes by the name watson and I listened to their entire catalogue and made a playlist of my favourites to be able to easily select a piece that fits any game I'm working on. Their umbrella website MusMus is https://musmus.main.jp and whilst you'll need to run browser translation on the site you will see all their music sorted across many categories. That said, it is definitely easier to listen to their music through the albums they've released on streaming platforms: Spotify and Apple Music for certain, and probably others. They're also on YouTube.

Of course, there are many more Japanese services and people writing great royalty free music. Another of my games used royalty free music by a young Japanese musician named YuyakeMonster, again picked specifically to fit a particular game: https://soundcloud.com/mac-vogelsang/sets/sparrow-solitaire

palemoonale2• 3 years ago

I thought about for long how SFX in f.e. airsims are done nowadays: different engine characteristics are just one thing. Are there still samples / recordings involved?

Same with racing or train sims.

msephton4• 3 years ago

I expect so, but it would depend on the game. I know they still use engine samples (and probably hugely advanced systems for playing them back) in car racing games like Gran Turismo.

pengaru2• 3 years ago

You might find this software interesting:

https://www.youtube.com/watch?v=RKT-sKtR970

I don't know if any of the state of the art flight/driving simulators are doing this kind of engine noise synthesis already, but they should be.

palemoonale2• 3 years ago

Thanks!

meheleventyone2• 3 years ago

The problem here is mostly computational cost. Simulating finer grained details needs a proportionally higher tick rate. In this case although the engine sim seems to be 2D and the gas sim 1D but both would need really high resolution to output sound synthesis. Driving games for now are largely putting that work into tyre dynamics as it’s such an integral part of the driving experience. Maybe there’s head room to do this x number of cars on the track but I don’t think it’s a foregone conclusion that they can just slap in a similar simulation.

mortenjorck• 3 years ago

While not as procedurally advanced as some of the techniques explored here, in my opinion the absolute pinnacle of dynamic music in video games is the soundtrack to Mount Wario in Mario Kart 8.

While the level itself (an unusual point-to-point, highly vertical course) is its own masterclass in terms of game design, the music, comprising three separate movements, each with its own clever way of masking the transition to the next, is unparalleled in ramping up the excitement and drama in concert with the gameplay.

A short video on the music of Mount Wario: https://www.youtube.com/watch?v=-8TBS75U1p8

A longer deep-dive into the music theory of the score: https://www.youtube.com/watch?v=-6K4NiDP04I

msephton4• 3 years ago

Fantastic, Mount Wario is the greatest!

ronyeh• 3 years ago

Thanks for sharing. I learned something new! That’s my favorite level, and I love the crazy shortcut.

the-golden-one• 3 years ago

DirectMusic had aspects of this.

https://en.wikipedia.org/wiki/DirectMusic

jader201• 3 years ago

I just recently played Nier Automata for the first time, and was amazed by not only the soundtrack, but the way it handle dynamically changing as you interact with other characters, or when certain story elements are triggered.

There’s even an in-game jukebox that lets you explore the music, and it’s crazy to see how nested it gets based on different environments.

Definitely recommend checking out if you, like me, are elevated by games with great music.

jsomedon• 3 years ago

I was just about to mention Nier Automata too. The dynamically changing music was a huge wow moment to me when I played it.

Actually this game is such an art. Its story is touching, full of unexpected plot twist, and inspiring deep thinking afterwards; its graphics is cozy and unsettling, heart warming and hear breaking at the same time - yet the game is not really gpu demanding; its combat is slick, snap, and satisfying; its music literally echo with your emotional roller coaster along your gameplay. And what's crazy is that, all of these nice things about this game actually blend with each other really perfect, so you don't think about any single aspect of this game, you feel the whole game as a package delivers you one of, if not THE, finest japanese anime/game experience in so many levels. I was really depressed for like 6 months after finishing this game because I left wanting more of Nier Automata games. It will literally make you become a huuuuge weeb.

jasomill• 3 years ago

I especially like the seamless transitions between different orchestrations of the score when moving between certain areas, and the chiptune transition when entering the hacking minigame, e.g., from

https://archive.org/details/nier-automata-original-soundtrac...

https://archive.org/details/NieR_Automata_Original_Soundtrac...

kyleashipley• 3 years ago

They wrote up a pretty good blog post about how they made the music transitions work: https://www.platinumgames.com/official-blog/article/9581

Definitely on my list of games I wish I could play again for the first time.

nocsi• 3 years ago

Super Crazy Rhythm Castle just came out and it’s entirely about procedural sound and music.

snailmailman• 3 years ago

Reminds me a bit of a game dev talk from a Valve employee. Having trouble finding the video, but one of the interesting things that stood out to me was all the challenges with sound effects in VR. For a door, you can’t just have “door opens” being a simple creak. Because in VR the user physically drags the door open. They can drag it part way, they can open the door quickly, slowly, etc. What in other games was a simple audio file became very complex in VR.

c-hendricks2• 3 years ago

That might be from their hour-long talk on Doors: https://m.youtube.com/watch?v=9kzu2Y33yKM

rkagerer• 3 years ago

I've noticed while playing PC and Android games, volume will sometimes apparently increase or decrease on it's own - without me touching the volume knob or any visible change in the volume setting. Is there some known bug in common (eg. Realtek?) drivers or hardware that may explain it? Has anyone else encountered?

sk0g• 3 years ago

Is some sort of volume "equalisation" or compression being applied maybe? Could also be reacting to the change in ambient volume conditions otherwise.

tuzemec• 3 years ago

I remember a talk by Borislav Slavov (composer for Divinity: Original Sin 2, BG3) where he was showing how the music changes depending on the player's actions in D:OS2.

One of the examples, that I found really cool, was switching from major to minor scale when the player enters in stealth mode. There were other interesting bits and tricks, but this one really stood out for me.

Also, it worths mentioning that the player was able to choose the lead instrument for the in-game music during the character creation - flute, tambour, oud, and cello.

Not sure if that talk is available online. I've got the chance to meet him in person a few years ago.

wirelesspotat• 3 years ago

Reminds me that there's some interesting audio techniques in the new Avatar game, like ray traced audio:

https://youtu.be/LRI_qgVSwMY (skip to 13:16 for the audio section)

c-hendricks2• 3 years ago

Sonic Frontiers had a fun take on the music for boss battles, with each song being broken up into not-so-obvious loops, with the action masking transitions so that each phase of the boss fight always goes with a specific portion of the song. If you can lean into the cheesiness it's very fun.

derpitron• 3 years ago

Tangential to this topic, my favorite example of dynamic music is in Need for Speed Most Wanted 2005's police pursuit soundtrack: https://youtu.be/YtSQCYStwew

While certainly much more primitive than modern games (it's merely looping segments of a premade track), Most Wanted 05 does a great job of live scoring the pursuits, playing, looping, and smoothly segueing between the track segments. From experience I guess it selects the segments based on the heat level, player speed, police proximity, and other factors that I'm not even sure about.

There are 4 pursuit soundtracks in the game, I highly recommend people listen to them if not try the pursuit mode itself in game. If nothing else it's thrilling as hell, a feeling that the modern entries still somehow fail to capture fully.

Dynamic music and sound techniques for video games

Related Stories