General soundtrack stuff

[Lucifer]

So I'm thinking of a class that will load sound effects, and another class that will handle music playback, and I've got some random thoughts to throw out.

Is it ok to marry ourselves to SDL_Mixer? The code currently supports sound without it, but moving on, we really can't do a lot without something more advanced, and SDL_Mixer is the more advanced something we need.

I've looked over how the cycle effect in particular is handled, but I'm a tad confused. Where and how does it get handled within the game loop?

So I'm wanting to create a series of mappings, like maybe as enums or something. They'll be things like "CYCLE_ENGINE", "CYCLE_EXPLOSION", "GRIND_CYCLE_WALL", "GRIND_GRID_WALL", "ENTER_WINZONE". Most of them will be empty and not contain any sound files (until someone creates the sound effects for them). Then in the game loop, when it's time to trigger an effect, we'll call a static method on the player, something like eSoundPlayer::PlayEffect(CYCLE_EXPLOSION). Then SDL_Mixer will be told to fire off the CYCLE_EXPLOSION. We could pass some other parameters, like the player's position and the position where the sound originates, and the soundplayer can use that to determine how to play the sound (mix for 3d sound).

The soundplayer could then load a sound effect skin file that makes the mapping to actual sound file on disc conversion. This should be an xml file that follows the resource model given by the maps, and it will be loaded from the resource manager. So, next question coming.

The individual sound files that go in a sound effect skin should be stuck in an archive with the xml file that describes it. The resource manager should be able to retrieve that archive. Should it also be able to retrieve each file individually? So the sound effect skin could specify the urls for every file, and the resource manager could retrieve them one by one. Should it? Or should we just require a sound effect skin to be a complete archive of a given format (.tar.gz or .zip)? If we opt for the second solution, then do we need to use the resource manager to load each individual sound file, or should we use it to load just the skin descriptor file and then load the sound files ourselves in our own peculiar fashion?

I'm thinking about how to structure the music track. I like the idea of having a series of ogg files linked together as a song (for a primitive engine, that is), and then letting the game randomly select which ones to play at a given point in time. The end result of that is either the song will play indefinitely, or we need some way to specify that *this* song ends after 5 minutes, and you should start a new song. The obvious solution is to just make each song a complete self-contained file, and that's probably what it'll look like in the initial implementation. Maybe that'll be good enough?

So when the game starts, I'm wanting a nice, dramatic theme to play. Then I want that theme stated in a regular song, with a beat, and bass, and all that good stuff. This part I want to have play indefinitely until a round starts, so when the player joins a server and a round is in progress, the in-game portion of the music track won't start until the new round starts and the player starts playing.

So, when the splash screen is displayed, the title track will play. Say it's about 15 seconds in length. The user will interact with the application just like he does currently, we won't make him wait those 15 seconds. We could make him wait 3-4 seconds and have him watch a cool animation, but that's beyond the scope of a soundtrack.

So then the game will load the gui music track and repeat it until a round starts.

When a round starts, the game should do something based on the player's preference. Either it will play it's own in-game stuff, or it will play music the player has asked for. So we have several possibilities:

* The in-game soundtrack as I've described.
* Load an m3u playlist that contains complete self-contained songs that we've bundled with the game, and then just do a randomize on it.
* Load an m3u playlist that the user has built with his own stuff and do a randomize on that.
* Play a CD that's in the cdrom drive.

When the player leaves the game, fade out whatever's playing and go back to the gui music track. When the game shuts down, fade out whatever's playing and unload and free memory and crap.

So the eSoundPlayer class I'm envisioning should handle all sound effects and the music track (divorce this from any existing eSoundPlayer class that may exist). You call it to say "Play this effect". Or you can pass it events, like ROUND_START, MATCH_END, PLAYER_VICTORY, PLAYER_LOST, THIS_PLAYER_SUCKS. Then the eSoundPlayer class would play the appropriate tunes to go with it, as well as any special effects that go with it. So in an event queue, we need the eSoundPlayer to receive events. Does it need to send events? I can't think of a reason it would need to.

So it sits watch-dog over the game waiting for something to happen. Does it belong in its own thread? It has to manage tasks that aren't related to the game loop. It has to manage SDL_Mixer, specifically. When a song ends, it has to decide what to do next, either pick a random song to play, or just fart. Or both. But this is independent of the game loop. Should it be an updateable game object that gets called from the game loop to update, or a separate thread that updates itself as needed and sleeps the rest of the time? SDL_Mixer does the hard work of playing the music, so our sound player will sleep most of the time, making most update calls a quick if(asleep) return;, which shouldn't impact performance severely. On the other hand, sticking it in a thread means it only impacts performance when it needs to, and we let the kernel deal with it.

[Z-Man]

The music thoughts are sensible, SDL_Mixer has the appropriate callback mechanisms. You can get yourself notified if the music play ends, you can fade out the music and be notified when that finishes. Unfortunately, you're not directly allowed to start the next track in this callback, you have to set some bit somewhere and poll for that in the main loop, or send an event.

Sound is not processed at all in the main loop, it works differently than graphics. You don't render sound frames and send them to the sound card one after another. Instead, the sound card sends interrupts whenever it wants new data, and then you need to supply it. In SDL(_Mixer), you register a callback for that. This callback is always executed in another thread.

I'm aftaid one global sound player is not enough, you can't handle the continuous motor noises properly. We'll need, in addition, game sound objects. A game sound object would be something with position and velocity that knows how to mix its sound effects relative to the current camera/microphone position. Right now, the eGameObjects handle it themselves, but we already said that's a bad thing when it comes to rendering, so it's a bad thing with sound, too. So, an abstract eSoundPlayer class is needed that just mixes in a sound. In addition to a (maybe singleton) global instance, we need a eSoundPlayer3D with position and velocity, and eMicrophone3D with position and velocity of the camera, and eSoundPlayer3D needs to hanle the mixing. Much of this is already there, it just needs a bit of reshuffling. And 3D effects need to be implemented, like a doppler effect, for example.

Is there a chance we could support true 3D sound? The stuff with more than two speakers?

Oh yeah, the sound intensity calculation needs reworking. Sounds get too soft too fast when you move away from the source. The effect is physically correct, but as is often the case, does not work well.

[Lucifer]

I think we have true 3d sound in SDL_Mixer. There's a function to play a sound file that takes distance and bearing, anyway. I assume it works for all output channels. If not, it's fairly straightforward to add an effects processor to mix into 4 channels or whatever.

[Lucifer]

Ok, so SDL_Mixer has these things called "channels". I assume you've seen them? They're not linked to any channels on your sound card, they're more like channels on a mixer (hence the name, SDL_Mixer). So you decide to play a machine gun sample, you tell sdl_mixer to play this sound in this channel, and sdl_mixer will mix it in with any other sounds currently playing and also the soundtrack, to produce a 2-channel (or more, depending on your sound card) output.

So the sound player needs to take a call from somewhere in the game that says "Play this sound", find an available channel, and play it. It generally doesn't need to worry about mixing it with the other sounds playing, and it doesn't need to worry about a lot of things. If it needs to apply an effect (such as an echo or what-have-you), then it uses a callback to do that, and SDL_Mixer calls the callback with actual sample data.

So, we *could* make a base class, your eMicrophone3D class that holds camera data. Seems like it would be attached to the camera anyway, so is there any reason we can't just get position and bearing of the camera directly? For all sound effects other than the cycle motor, we can pass position and intensity of the sound effect. If we want to get real exact, we can also pass it a vector that represents the direction of the sound, and some modifiers (for that sound modeling equation, I forget what it is, we just covered it in trig). I'm inclined to just take position and intensity from the gme objct, compute the angle and send all that junk to sdl_mixer and let sdl do the rest. Anyway, my point is, we will need helper classes of some sorts, but we don't need to do any mixing, all we need to do is figure out where the thing is in relation to us. I'm still feeling pretty half-baked on this, I must admit. I apologize if I'm just restating stuff you already said.

http://jcatki.no-ip.org/SDL_mixer/SDL_m ... html#SEC76

The cycle motor is throwing me for a loop. All the other effects are easy, just push the button to play the effect. We could have each game object give up a callback of its own to mix continuous sounds based on game conditions, but that puts audio code into game objects, which you just said we shouldn't do (and already do). Seems like we need a class that's eContinuousSoundEffect or something like that, and it contains the processor, whatever it is (resample to speed up or slow down). Then somewhere in the meta data for the game objects themselves, we'll want to associate the game object with an instance of the eContinuousSoundEffect. Then the eCSE class can poll the game object for the data it needs. We don't have to worry about mutexes and crap (polling from a separate thread) because we're just reading position and velocity from the game object, and it only has to be accurate within a few timesteps anyway. Does that work? It feels right...

I'm going to start by implementing the eSoundPlayer class as a music player. I'll probably stub some of the sound effect playing stuff, but I'm intending to start it as a music player. That's the lowest hanging fruit. I'll probably go grab some free mod files to test the code with, since I'm still drawing a blank for sound track music. Maybe Seeker will produce something and I can just use that. But there's a bit of a name collision. There's already an eSoundPlayer class, and I'm drawing a blank on what to name the new one. I'll come up with something, I guess.

So I figure in program initialization, we're going to need to poll objects that aren't even created yet and find out what resources they need so we can make sure we have them. This isn't just sound. I'm inclined to load all sound, textures, and models into memory during program initialization, and then not load them again after that except by some reasonable user command to reload (such as changing skins). I'm not in a hurry to figure out how this should be done yet (seems pretty obvious, I guess), just thought I'd throw it out there. When we make it possible for players to have custom cycle models, it'll be logical to say they should also have custom sound effects to go with the model.

(On the other things I had going on, if you're wondering "Why is he starting something else when he hasn't finished what he started already". The config system can be pulled out, it's still completely half-baked. I've let it cook for awhile now and there are some issues with it I just haven't been able to resolve, leading me to conclude it's fundamentally flawed. I can't think of any reason to leave it there like it is, and if there are no objections, I'll pull it myself sometime. The particle system, hmmm, I'm stuck there. I'm hoping 0.2.8 will get released soon and someone else will take a look and either unstuck me, or just finish it. So I'm working on the particle system by hope. And I'm intending to get the web server working to serve resources over christmas break. Not sure if I'm going to after getting CGI working on it or not, but I don't want to be in a position where y'all pull a 0.3 release on me and I have to rush up to get it to serve resources. So I'm still juggling some of them )

[Lucifer]

Thought about the cycle sound some more.

I don't know if there's an upper bound on the number of channels available in sdl_mixer, but it seems like we should give each cycle a separate channel for its speed. Then we set the channel to loop and give it an effect callback, and that effect callback will speed up or slow down the sample with the cycle's velocity. I'm drawing a blank on how to plug it in, though. sdl_mixer is a C library, so we need c-style function signatures for the callbacks. It'll make sense when it's time to do it, I think.

I'm thinking we need at least two paramters for this sound. One to modify the tempo of the sound (for lack of a better word), and one for its frequency. So you could have, say, a closed hi hat sound for the motor. Instead of increasing its frequency with the cycle's speed, you might just want it to repeat faster. Or you could take for the motor sound something like a heavily distorted guitar playing a low E power chord and change its frequency (I've done that, at least, and it sounds really cool). Or you could do both.

I want to record my truck's engine and use it for the cycle motor sound.

Ok, so with the cycle motor, here's where it gets complicated. I'd like to be able to have an arbitrary number of sound effects related to the cycle's velocity. So maybe we want to have the whooshing sound of a fast-moving object in air, and also a motor sound. Then we want to add the tire-slapping sound on top of all that. That's three different sounds with potentially different effect requirements. The whoosh should increase in intensity with your distance, the tireslapping should be the aforementioned increase in tempo but not frequency, nd the motor might be a change in frequency. Then you have 16 cycles on the grid, so 48 channels just for cycle noises! That's a lot of mixing, a slower computer will have trouble with that. Maybe if it's possible we can code it in, but our default sound skin won't do all that work.

[Lucifer]

Heh, still thinking about it (and working on it a bit).

So, when you register an effect callback, you can send it a pointer to some arbitrary piece of data. You can register the same effect callback with different pointers as many times as you want, so we can have one single effect callback that we register with pointers to gCycles, and then just read position and stuff from that. Right? Still no idea how to get the camera, but mostly that's because I've never looked at the camera code. Is the camera available from gCycle? Should it be?

[Raoul Duke]

I think it would be a great idea if the music was tron user made

/me finds winking smilie

can't find. I'll go with semicolon parenthese.

[Lucifer]

What a waste. All that work trying to figure out why a song won't play, only to find that SDL_mixer is broke on my computer.

[Seeker]

I started working on a few songs but I want some info before I get finished.

How long of a song do we want.........epic peices or sound clips?????

What format or have you come up with that yet?

Im focusing on industrial and techno for my stuff..........for now

[Lucifer]

I was going to work up some rock/metal stuff. We definitely should collaborate at some point in the future, but let's get some stuff together first.

I'd like to offer sound skins that include replacements for stock stuff. So I guess you should make a title track, gui track, and some stuff for in-game music.

The title track + gui track should essentially be your standard "intro - song" setup, where the "song" part needs to loop cleanly. So about 10-20 seconds of length for the title track, but the gui track should be built on the same theme.

For now, I'm just going to work on complete song support through an m3u playlist. Randomized. So make sure that each song loop cleanly and smoothly transitions into each other (not terribly difficult), and that they can be played in any order without disrupting anything.

Later on we'll look at splitting them up into smaller pieces and using in-game heuristics to assemble the tracks to match the action better.

I have title track working right now, the rest should just fall together. No new sound effects, though. Once I get a gui track (that will loop endlessly for now) I'll add sound effects, all except the cycle motor, and get that all working, then commit. I'm working with cvs head, it probably goes without saying. It's fairly straightforward to backport to make an 0.2.8.1 if we want, you know, sometime after we've made 0.2.8.

[Lucifer]

Sometimes I just really suck.

Anyway, leaving that cryptic statement unexplained (heh), I documented it on the wiki.

http://wiki.armagetronad.net/index.php/New_Sound_Engine

The upshot is that there's music, new settings items that are likely to disappear after awhile (before 0.3, ideally, but not necessarily), and there's no sound effects.

Left this off the wiki, I'll add it when I wake up:

This is only available in Linux right now, and you get it by configuring --enable-music. In Windows you'll have to do some extra work, and I don't know about Mac OS X. If you're working on one of those platforms, and the build has broken because of this, just do whatever you need to to make it work again, unless you actually want to work on the new sound engine. Otherwise, it'll probably be time better spent to wait until it's ready to be brought up and running on the other platforms.

Edit: Oh yeah, the whole thread vs polling in the game loop confusion disappeared when I remembered we had per-frame tasks. So the sound engine's update() method is called each frame, and only updates when something's changed. Which brings into mind one important issue, which is the fadeout. If you call Mix_FadeoutMusic() (or whatever that function is called), it blocks. So if you want to fade out over 2 seconds, you're blocked for 2 seconds. Anybody got a workaround for that? I don't, not without threading. Heh.

[Z-Man]

The reason for eMicrophone would be: a case can be made that the current camera position, direction and velocity are not the correct reference points for sound. Rather, your cycle should be it. You want to hear the doppler whoosh when another cycle passes your cycle, not when it passes the camera. You don't want sound to get out of control when the smart camera zips around.
Best is probably to take position and velocity from the cycle, but direction from the camera. In Gothic (a third person action RPG), we did something even stranger: the sound intensity was determined by the distance of the source to the character, but the direction the sound appeared to come from was determined by the angle from the camera to the character.

There probably were other things I wanted to comment on, but I forgot. Lucifer is posting faster than I can think, and the thoughts timed out

[Lucifer]

Hmmm, seems like if we take position and velocity from the cycle, we should also take the cycle's heading. Is it believable to see smartcam roaming around the cycle, but the sounds you here are intended to be what you'd here if you were in the cockpit of the cycle? I suspect it is...

I've just about got it figured out, then, and partially coded. I'll dd a method to eSoundMixer, SetMicrophone or YourEarsAreOwnedBy, which will take an eGameObject*. Then it'll poll the eGameObject for position, velocity, and direction when doing any computations that depend on these things. When you switch game objects (like when you're dead and watching other players), then this method will need to be called to update the mixer.

Seems like we should have a generic DopplerShift effect that each sound effect has to go through, but we can worry about that later.

So I'll make PushButton take a position, velocity, and heading as well, but first I'll only use position to make the sound effect.

So a question, then. I've got explosions working, but I did it using gCycle::Kill(). I noticed that when the round ends, all cycles get killed. So the explosion effect is being sounded for all cycles. Where should I be triggering the explosion effect? (heh, I looked around quite a bit for it)

Also, it looks like the cycle turn wav is used for a couple of things? Is that correct? The new system will split all of them up (and add some more sounds, courtesy of voltsubito), so if you want to use a sound for several things, you'll have to render it appropriately to those things. Um, in the meantime, should I just create new sounds trying to get as close to the old ones as possible, or do you have a neat trick to render them to disk so I can use exactly what's there?

[Lucifer]

Ok, made a SetMicrophoneOwner method. Now I'm looking where to call it when the round starts. I found where to call it when the camera switches position after you die, but still not where to call it for the current player. A slight complication that's already solved in the code (or rather, I think it has to be, so I ssume it's there), who owns the microphone when there's 2 or more players on the same computer?

I'm ready to make explosions have a 3d effect, but I still need that. I'll keep looking.

We need more sound effects! Voltsubito was kind enough to give us 3,2, and 1, but no "GO!". So it counts "3", "2", "1", and then nothing when the round starts. If we stick with voltsubito's voice for the announcer, then he kinda needs to show up again and give up more. Otherwise, we'll need to start fresh with these sounds:

Announcer says 3, 2, 1, and possibly "go". We can use an abstract beep instead.
Time warning! We have five minutes left, two minutes left, 30 seconds left, etc. All of them. I'll make a more comprehensive list on the wiki.
Round start, round winner, match start, match winner. These can all be abstract sounds, and we can leave them blank for the default distribution and let extenders add more to their own packs.

Personally, I'd prefer a female contralto with a british accent for the announcer. Which means just about any European voice will do, provided the native accent isn't too thick. If the native accent is fairly thick, then my own preferences are Spanish, followed by any Germanic/slavic accent. Naturally this brings up the issue of localising sound effects. Heh. No idea how big it'll make the download to localise sound effects just for the languages we support now, but it'll definitely make a difference when we support a bunch of languages. Any ideas how we want to deal with this? Just let the download grow in size? Split up localised sound effects into a separate package(s)?

Not sure what else right now. There's plenty of room to add more. It's pretty extensible now.

Working on getting directional effects to one-off sounds, then I'll look at cycle motor sounds. I'll be committing with voltusbito's announcer samples, we can update them later if we need to. They sound pretty decent.

[Lucifer]

Ok, I'm definitely stuck. How do I find out, at the start of a round, which camera belongs to the player? And how should I differentiate between players on the same box?

I found uActionPlayer and traced a few paths through it, but I'm still kinda confused. Anyway, it looks like uActionPlayer is the player who gets credited with things like chat and stuff, so I want to know who uActionPlayer is. So going through uActionPlayer's static methods, how do I get to a eGameObject* that represents that player?