Many are still looking for a solution for the Silverlight sound problem. What are the problems and what is the solution?

People who tried to create a casual game in Silverlight, encountered problems when they added sound effects to the game. The time between starting a sound effect and actually hearing it, is unacceptably long. This is called “audio latency”.

Short effects, typically less than a second, sound good the first time, while they sound bad the second time when they are played. The source of this problem is not clear, however long sounds don’t have this problem. This is called the “short sound problem”.

Also, it is very hard in Silverlight to play music in an idle loop. An idle loop with the out-of-the-box MediaElement, plays a silence every time the music is repeated. The silence at the beginning is actually caused by the same audio latency described above.

The above problems exist in Silverlight 4 and lower versions.

The audio gap
The music compression format MP3, is standard encoded with a silence at the beginning – the audio gap. This is because the decompression code needs some time to get ready, like an engine which needs time to get started. Tools like WinLame may encode an MP3 without an audio gap at the beginning, but this does no good to Silverlight’s decompression code. An MP3 created with WinLame will sound bad.

The audio buffer
Silverlight uses an audio buffer internally, which needs to be filled with audio data before any sound is played. The size of the audio buffer is measured in time, and it costs exactly that time to fill the audio buffer, before any sound is heard.

Reducing audio latency
The audio latency is caused by the audio gap from the music format and because of the audio buffer. So if we want to decrease the time between calling the Play function and actually hearing sound, we need to handle both the audio gap and audio buffer. Now obviously, a latency of 0.0 is impossible, otherwise Microsoft would have provided this with Silverlight.

The solution: VorbisPlayer
The solution is to choose for an audio format which is gapless. Second, the default length of the audio buffer must be decreased. And for a seamless idle loop, we must stream the next music part into the audio buffer, right after the previous music part has finished.

Ogg Vorbis is a music format which can be encoded without audio gap, however it is not supported by Silverlight. However, Silverlight does allow developers to implement their own audio decoder.

The VorbisPlayer provides exactly the solutions described above. More important, the VorbisPlayer is easy to use, for not so technical people.

The VorbisPlayer allows low latency for sound effects, much lower than Silverlight provides out of the box. Because of the used technique, the short sound problem is also solved. Also, the VorbisPlayer has an easy function to invoke a playlist of music, which will be played seamless without gaps in between.

The Silverlight VorbisPlayer can be found on Codeplex: http://vorbisplayer.codeplex.com/

Great developments on the VorbisPlayer. To reduce the download time of Slengo, the bitrate of all sound in the game was reduced from 44.1Khz to 16Kh. Although the quality of the sound is lower, it still sounds quite good. 1.87Mb of data was reduced to 0.99Mb.

Another advantage is that the VorbisPlayer uses less CPU at this bitrate, and performs much better. On high-end machines, people did not have much problems with performance, however on lower end machines the sound could stutter. With a lower bitrate, this problem solved for low-end machines. The VorbisPlayer was modified to allow for lower bitrates.

A new release of the VorbisPlayer has been uploaded on CodePlex:

I have made some progression since the last time I wrote an article about my new game Slengo.

 
I created a Vorbis player UserControl, optimized for sounds effects in games. When initialized, the MediaElement inside will be streaming silence. When the function Play is called with a stream, the silence will be replaced by this stream, until the end of the sound, where the stream will go back into silence mode.

This scheme solves the short-sound problem, ie MediaElement is incapable of handling sounds shorter than 1 second. The silence-stream actually makes the sound effect longer than a second. Secondly, this scheme solves a major part of the sound-latency or audio-gap. With a buffer of 45ms, this latency is drastically lower than what you get out of the box. I develop in a Virtual Machine and I’ve been really pushing to get this latency as low as possible. Then I tested on iron, which demonstrated a great latency improvement.

The only drawback of this scheme is that you cannot make major sound changes while continues playing a stream. It is not possible to alter the audio buffer length, it is not possible to switch from mono sounds to stereo sounds. Those are acceptable limits.

Past week I have added another NPC to the game. And ofcourse tested her (yes it’s a she). That currently makes a total of four NPC’s (enemies).

This weekend I took my Vorbis player to the next step: seamless audio. So you buy a couple of loop-sets, encode them to Ogg Vorbis and just mix the tracks. A MediaEnd event is triggered when a sound has finished playing, and a tracker function just injects the next track. As there is no audio-gap, the tracks play seamlessly after eachother. And at the end of the playlist, you just repeat the whole thing.

The next step was to add the tracker to the game, so that you hear the music and the sound effects at the same time.

A weird thing happened. While in my test-tracker, the background music played without a glitch, in the game the sound began to stutter and distort. I thought that the decoding routines were slow, so I thought I decode all sounds effects at the start of the game and keep the result in a cache. Only the music would then be decoded at playing time. Well, this caching scheme did not work, the music still sounded distorted. The solution was to use less VorbisPlayers for the sound effects. Apparently, there were too many MediaElements active at the same time. That’s one of those cases where it is a good thing getting to know your tools.

I’ve also worked on a title screen for the game. I’m keeping it simple this time. I also want to introduce a level picker, where a level is unlocked when you reach a certain level. That sounds easy. The hard part will be next, creating challenging levels, which keep game-players attracted.

Update 11 jun 2010.

For those interested, the VorbisPlayer has now been published on Codeplex. Find the source code here.

----

I have finished the playing part of Slengo. Next are the titlescreen, the highscore, making some levels and… adding sound effects. When I started adding the sound effects, I instantly remembered why I use little effects in my games: the slow latency.

It is like this. On the screen you push an icecube, and half a second later (or longer) you hear the sound effect. The visual and audio are so much out of sync that it breaks game-playing experience.

In Avios, I solved this problem by delaying the graphics. The event of a bomb falling into the sea works with this principle. This was possible because the sounds in Avios are not bound to user-events. They are bound to in-game events. With user-bound events, this has the drawback that the game will appear to react slowly on user input.

The delay has two causes.

1) Mp3’s are standard encoded with a gap at the beginning;
2) The MediaElement has an internal buffer which is filled before even one sound is played.

Also Silverlight has problems with short sounds (less than a second), they make the delay worse, or you don’t hear them at all.

Countering the audio gap
I have thought of two ways to deal with the gap, encoded in audio files. Either find a gapless format or gapless encoder, or skip the gap when starting the sound.
If you encode Mp3’s gapless with WinLame, then the audio quality is very bad (I’ve tested a short sound).

In Silverlight, it is possible to write your own sound decoder. Larry Olson wrote Managed Media Helpers which inspired about everybody else. What I understand from this project is that only the Mp3 frame headers are decoded, while the compressed sound data is send to the MediaElement, making use of the decoder already in Silverlight. If this is true, then this scheme is not sufficient to skip the gap programmatically, because you cannot set a start pointer in the middle of compressed data.

DDtMM wrote a library for seamless audio loop, for Mp4 encoded files. Expression Blend Encoder can encode gapless. I had many troubles downloading the library in the past (timeouts), so I was unable to investigate this code. Recently I had more luck, but being too occupied with MoonVorbis (see below) I haven’t studied the code very well yet.
Ahura Mazda wrote the Saluse Media Kit, fully decoding Mp3’s and sending the raw audio to the MediaElement. This code is a good option to skip the gap at raw-sound level. Unfortunately I had problems playing short sounds repeatedly, so I looked further.

A project which really drew my attention was MoonVorbis from Atsushi Eno. Casy Wireman wrote on GameDev that Ogg Vorbis is a pretty good format for games. It seems that Ogg Vorbis is encoded without audio gap. MoonVorbis decodes to raw sound samples. So if there is a gap we can skip it.

Countering the audio buffer

MoonVorbis is not production ready. I had problems with the pitch of my test-sound, and problems with repeatedly playing a short sound. However, I’ve worked a solution. But still, there is latency between user-event and sound being played.

I have introduced a Play function on the stream. The MediaElement’s source is set to the stream and starts playing directly. The stream produces silence when the sound is not played. When the Play function is called on the stream, it starts to send sound samples. With this approach, the short-sound problem is solved, because the standard silence extends the length of your sample.

Now to reduce audio buffer latency. The MediaStreamSource has a property called AudioBufferLength, which is measured in milliseconds. The standard setting is 1000 (1 second), making the latency very slow. If you set this property to its minimum, 15 milliseconds, the latency is much better, but the sound begins to stutter. A setting of 50ms gives a little more latency, but with good sound quality. I am still investigating whether this is the best we can get.

Further

Solving the short-sound problem is a major improvement. I would still like to reduce the sound latency a little more. However, I still have the option of delaying the graphics, but now for a much lower latency. That is an improvement. Still, I need to find out how all this works out in practice when I bring it to the game. Although I have a preference for WMA audio, Ogg Vorbis is a good enough choice for me.

Avios is a great game if you liked Wings of Fury during the nineties. It took 5 months to develop, but it isn’t very well appreciated. There are different aspects for improvement. One aspect is to get a better loading time experience.

Players of online games don’t like to wait very long for a game to load. I have seen different approaches for better a better game-loading experience.

1. Load everything all in one
The game exists of only one xap, which is loaded at once. At Mashooo and Silverarcade they’ve found a way to improve experience. Each game is loaded by a generic preloader, displaying the current percentage. This is very well done. The drawback is however that the game will be loaded completely, you cannot split it into parts. This approach cannot distract the player’s attention if the game is very large.

2. Partial preload
A preloader loads the basic data. The rest of the data is loaded while the game is playing. This is the scheme I’ve been using for most of my games. In my case, the preloader has a list of images and sound effects to be loaded. Any music which is played during the game, is played from the server during the game. Music files can be quite large and they’re not always played completely. So this approach has an advantage over loading it all at once.
You also have more control of what will be loaded in the preloaded, and what will be loaded during the game. Nokola’s Shock for example loads the most basic things to play the first level in the preloader and everything else during the game. Avios however loads all images and sound effects for all levels in the preloader.
An improvement for Avios could be to load everything to play the first level, while loading everything else when the first level is played. No problem for Avios. But in a different game, in such a scheme, the loading and playing of music could compete with loading other data. It could interfere with the music being played. So this needs to be tested.

3. Lower quality
It isn’t surprising that visuals in many games are very small. Smaller graphics reduce the data volume and give better loading experience. For the same reason, players experience lower sound quality for the same reason. Lower quality improves game load experience but may reduce playing experience. This balance is the choice.

4. Change the Encoding
Unfortunately, using vector graphics doesn’t increase quality, while data volume is reduced. I have been playing around with Inkscape 1), to convert bitmap images into vector graphics, exporting it in XAML. In many cases, where I replaced a bitmap image for a vectorized version, the data volume increased. If you want to preload such resources, you need to dynamically load a dll, instead of loading a list of resources on the server.

For my games, I usually use PNG’s. PNG supports alpha channel –  transparency. JPG’s do not support this. In Photoshop you can export JPG’s at low quality. One could use these low-quality JPG’s, assign a transparency color to it and process the images using WriteableBitmap, to support alpha channel. I have done a little study to this option, but I found the image-quality so very low that I decided it was not worth it.

There is also a gain for sound encoding. MP3’s are usually larger than WMA’s. So it does pay to encode for WMA. It is worth it to play around with the encoding settings. I was able to encode an original MP3 file of 1.8 MB, to a 491 KB WMA file, without significant loss of quality.