Many are still looking for a solution for the Silverlight sound problem. What are the problems and what is the solution?

People who tried to create a casual game in Silverlight, encountered problems when they added sound effects to the game. The time between starting a sound effect and actually hearing it, is unacceptably long. This is called “audio latency”.

Short effects, typically less than a second, sound good the first time, while they sound bad the second time when they are played. The source of this problem is not clear, however long sounds don’t have this problem. This is called the “short sound problem”.

Also, it is very hard in Silverlight to play music in an idle loop. An idle loop with the out-of-the-box MediaElement, plays a silence every time the music is repeated. The silence at the beginning is actually caused by the same audio latency described above.

The above problems exist in Silverlight 4 and lower versions.

The audio gap
The music compression format MP3, is standard encoded with a silence at the beginning – the audio gap. This is because the decompression code needs some time to get ready, like an engine which needs time to get started. Tools like WinLame may encode an MP3 without an audio gap at the beginning, but this does no good to Silverlight’s decompression code. An MP3 created with WinLame will sound bad.

The audio buffer
Silverlight uses an audio buffer internally, which needs to be filled with audio data before any sound is played. The size of the audio buffer is measured in time, and it costs exactly that time to fill the audio buffer, before any sound is heard.

Reducing audio latency
The audio latency is caused by the audio gap from the music format and because of the audio buffer. So if we want to decrease the time between calling the Play function and actually hearing sound, we need to handle both the audio gap and audio buffer. Now obviously, a latency of 0.0 is impossible, otherwise Microsoft would have provided this with Silverlight.

The solution: VorbisPlayer
The solution is to choose for an audio format which is gapless. Second, the default length of the audio buffer must be decreased. And for a seamless idle loop, we must stream the next music part into the audio buffer, right after the previous music part has finished.

Ogg Vorbis is a music format which can be encoded without audio gap, however it is not supported by Silverlight. However, Silverlight does allow developers to implement their own audio decoder.

The VorbisPlayer provides exactly the solutions described above. More important, the VorbisPlayer is easy to use, for not so technical people.

The VorbisPlayer allows low latency for sound effects, much lower than Silverlight provides out of the box. Because of the used technique, the short sound problem is also solved. Also, the VorbisPlayer has an easy function to invoke a playlist of music, which will be played seamless without gaps in between.

The Silverlight VorbisPlayer can be found on Codeplex: http://vorbisplayer.codeplex.com/

Great developments on the VorbisPlayer. To reduce the download time of Slengo, the bitrate of all sound in the game was reduced from 44.1Khz to 16Kh. Although the quality of the sound is lower, it still sounds quite good. 1.87Mb of data was reduced to 0.99Mb.

Another advantage is that the VorbisPlayer uses less CPU at this bitrate, and performs much better. On high-end machines, people did not have much problems with performance, however on lower end machines the sound could stutter. With a lower bitrate, this problem solved for low-end machines. The VorbisPlayer was modified to allow for lower bitrates.

A new release of the VorbisPlayer has been uploaded on CodePlex:

Update 11 jun 2010.

For those interested, the VorbisPlayer has now been published on Codeplex. Find the source code here.

----

I have finished the playing part of Slengo. Next are the titlescreen, the highscore, making some levels and… adding sound effects. When I started adding the sound effects, I instantly remembered why I use little effects in my games: the slow latency.

It is like this. On the screen you push an icecube, and half a second later (or longer) you hear the sound effect. The visual and audio are so much out of sync that it breaks game-playing experience.

In Avios, I solved this problem by delaying the graphics. The event of a bomb falling into the sea works with this principle. This was possible because the sounds in Avios are not bound to user-events. They are bound to in-game events. With user-bound events, this has the drawback that the game will appear to react slowly on user input.

The delay has two causes.

1) Mp3’s are standard encoded with a gap at the beginning;
2) The MediaElement has an internal buffer which is filled before even one sound is played.

Also Silverlight has problems with short sounds (less than a second), they make the delay worse, or you don’t hear them at all.

Countering the audio gap
I have thought of two ways to deal with the gap, encoded in audio files. Either find a gapless format or gapless encoder, or skip the gap when starting the sound.
If you encode Mp3’s gapless with WinLame, then the audio quality is very bad (I’ve tested a short sound).

In Silverlight, it is possible to write your own sound decoder. Larry Olson wrote Managed Media Helpers which inspired about everybody else. What I understand from this project is that only the Mp3 frame headers are decoded, while the compressed sound data is send to the MediaElement, making use of the decoder already in Silverlight. If this is true, then this scheme is not sufficient to skip the gap programmatically, because you cannot set a start pointer in the middle of compressed data.

DDtMM wrote a library for seamless audio loop, for Mp4 encoded files. Expression Blend Encoder can encode gapless. I had many troubles downloading the library in the past (timeouts), so I was unable to investigate this code. Recently I had more luck, but being too occupied with MoonVorbis (see below) I haven’t studied the code very well yet.
Ahura Mazda wrote the Saluse Media Kit, fully decoding Mp3’s and sending the raw audio to the MediaElement. This code is a good option to skip the gap at raw-sound level. Unfortunately I had problems playing short sounds repeatedly, so I looked further.

A project which really drew my attention was MoonVorbis from Atsushi Eno. Casy Wireman wrote on GameDev that Ogg Vorbis is a pretty good format for games. It seems that Ogg Vorbis is encoded without audio gap. MoonVorbis decodes to raw sound samples. So if there is a gap we can skip it.

Countering the audio buffer

MoonVorbis is not production ready. I had problems with the pitch of my test-sound, and problems with repeatedly playing a short sound. However, I’ve worked a solution. But still, there is latency between user-event and sound being played.

I have introduced a Play function on the stream. The MediaElement’s source is set to the stream and starts playing directly. The stream produces silence when the sound is not played. When the Play function is called on the stream, it starts to send sound samples. With this approach, the short-sound problem is solved, because the standard silence extends the length of your sample.

Now to reduce audio buffer latency. The MediaStreamSource has a property called AudioBufferLength, which is measured in milliseconds. The standard setting is 1000 (1 second), making the latency very slow. If you set this property to its minimum, 15 milliseconds, the latency is much better, but the sound begins to stutter. A setting of 50ms gives a little more latency, but with good sound quality. I am still investigating whether this is the best we can get.

Further

Solving the short-sound problem is a major improvement. I would still like to reduce the sound latency a little more. However, I still have the option of delaying the graphics, but now for a much lower latency. That is an improvement. Still, I need to find out how all this works out in practice when I bring it to the game. Although I have a preference for WMA audio, Ogg Vorbis is a good enough choice for me.