SourceDataLine Woes - Suggestions appreciated

[barfy]

I'm getting annoying clicks and pops with my sound playback using SourceDataLines (not using Clips since i want to manipulate sound data dynamically).

My initialization code looks like this:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22

// use a short, 100ms (1/10th sec) buffer        int bufferSize = playbackFormat.getFrameSize() * (int)(playbackFormat.getFrameRate() / 10);                          // create the buffer        byte[] buffer = new byte[bufferSize];                      // create, open, and start the line        SourceDataLine line;         DataLine.Info lineInfo = new DataLine.Info(SourceDataLine.class, playbackFormat);         try         {             line = (SourceDataLine)AudioSystem.getLine(lineInfo);             line.open(playbackFormat, bufferSize);         }         catch (LineUnavailableException ex)         {             // the line is unavailable - signal to end this thread            Thread.currentThread().interrupt();             return;         }         line.start();

And my playback (in a seperate Thread) looks like this:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15

while (numBytesRead != -1)                 {                    ....                                                // copy data                    numBytesRead =                         source.read(buffer, 0, buffer.length);                     if (numBytesRead != -1)                     {                           //line.write(buffer, 0, numBytesRead);                          int offset = 0;                           while(offset < numBytesRead)                                    offset+=line.write(buffer, offset, numBytesRead - offset);                                       }                 }

If I initialize line.open() with a LARGER buffer size, the problem is minimized. However, doing this results in latency problems and is especially apparent and ugly with continously looping sound effects. I'm guessing it's a buffer overflow problem wherein the data in the buffer gets overwritten before it can be used for playback. But then again, line.write() is supposed to block or at least supposed to check for available space before writing so I'm not 100% sure of the cause.

I'm wondering if anyone has got a solution for this?

I'd appreciate any suggestions.Thanks  

EDIT: I'm avoiding object creation overhead and sound latency by using a pre-defined number of Threads  (in a ThreadPool) initialized with Thread-local SourceDataLine objects and byte array buffers to play my sounds. When I limit the amount of Threads (and therefore the amount of reused SourceDataLine objects) used to a smaller number, the sound problems occur sooner. Calling line.drain() and line.stop() doesn't remedy the situation also. But if I just created new Threads and SourceDataLines each time to play the sound effects, the problems simply disappear... I wonder what's up.

[erikd]

I've had many problems like that, trying to get real time sound using a separate sound thread (although I'm not 100% sure if you are having the exactly the same problem).
In my case, the problem was caused by the fact that you can't be sure when the sound is updated when it's done in a separate thread of the sound playback thread.
I -kinda- resolved it at first by putting a small sleep in the playback thread (you could check to see if it helps), but it may result in buffer underflows sometimes.
I finally resolved it by getting rid of multi threaded sound rendering. I measure how long the last frame took, then I render exactly that amount of sound data.

[barfy]

Thanks for your input ErikD.

I've had many problems like that, trying to get real time sound using a separate sound thread (although I'm not 100% sure if you are having the exactly the same problem).
In my case, the problem was caused by the fact that you can't be sure when the sound is updated when it's done in a separate thread of the sound playback thread.
I -kinda- resolved it at first by putting a small sleep in the playback thread (you could check to see if it helps),

What do you mean by the statement "you can't be sure when the sound is updated"?

Are you also suggesting that I put a Thread.sleep() within the sound playback thread to artificially slow down the feeding of sound data into the SourceDataLine?

While that would be a hackish solution to a buffer overflow problem, I'm not sure that an overflow is precisely the problem since line.write() is supposed to block or at least not write if the space isn't available. On the other hand, it's not a buffer underflow issue as line.available() <= line.getBufferSize() always during runtime.

Btw, it sounds like your app used 1 main thread and 1 sound playback thread that spawns other threads for sound playback.

For mine, I basically have 1 main thread and up to 16 seperate sound playback threads (with SourceDataLines opened and buffers initialized) that are waiting on an object and are notified when there are sounds to be played. Some sound data are dynamically modified (just gradual volume change) during playback, but I'm also very sure that the sound anomalies aren't caused by this.

I measure how long the last frame took, then I render exactly that amount of sound data.

Sorry, could you explain that statement in a little more detail also? I'm kind of curious to see how you managed to implement simultaneous sound playback on a single Thread..  :-/

EDIT: I got around it by creating 2 thread-local SourceDataLines per Thread. 1 of them is used for normal sound playback and is opened() and initialized with a big buffer to eliminate those sound problems, the other has a small buffer to be used for looping sound effects and is not opened until it is ready to be used, after which it is closed() again. Very hackish, messy and sort of kills the purpose for a Thread pool (which is basically to minimize object creation and line initialization latency).

Arrgh! Grappling with javax.sound to make it work properly is needlessly tedious indeed...

Games published by our own members! Check 'em out!

[erikd]

Sorry, could you explain that statement in a little more detail also? I'm kind of curious to see how you managed to implement simultaneous sound playback on a single Thread..

Well, in the end all simultaneous sounds are mixed down to one sound stream one way or another, so I basically create that one stream right away, doing all mixing etc. myself. No need for threads.
I'm guessing the only reason you need threads for sound rendering if you expect them to block.
java.sound does block when the buffer is full. Now I write to the stream once a frame, and with exactly the correct amount of data needed for that frame, so I measure how long it took for the last frame, and calculate how much sound samples I need to render, taking the playback frequency into account (e.g. 44.1Khz or something). That way you can be sure you won't have buffer overflows and that there will be no blocking, so no need for threads. If you don't need threads, don't use them.
The stream I render is for all sounds in one go, mixing everything as I go.

Are you also suggesting that I put a Thread.sleep() within the sound playback thread to artificially slow down the feeding of sound data into the SourceDataLine?

It's not a real solution, but it worked (-ish) when my sound rendering was also multithreaded. The reason it worked for me was that the mean thread wrote sound commands to the sound thread (volume, pitch updating etc.). Because the sound thread was a tight loop, it seemed like those sound commands were not being written immediately, but were often delayed (and very irregularly, resulting in very weird timing errors). The sleep gave the sound command writing some more time (or so it seemed), and the timing errors went away.
But it's a very hackish solution to a problem that was basically caused by wrong design to begin with.

[barfy]

Well, in the end all simultaneous sounds are mixed down to one sound stream one way or another, so I basically create that one stream right away, doing all mixing etc. myself. No need for threads.
I'm guessing the only reason you need threads for sound rendering if you expect them to block.
java.sound does block when the buffer is full. Now I write to the stream once a frame, and with exactly the correct amount of data needed for that frame, so I measure how long it took for the last frame, and calculate how much sound samples I need to render, taking the playback frequency into account (e.g. 44.1Khz or something). That way you can be sure you won't have buffer overflows and that there will be no blocking, so no need for threads. If you don't need threads, don't use them.
The stream I render is for all sounds in one go, mixing everything as I go.

Hmm. Does it go something like this:

1. Get a single SourceDataLine from the AudioSystem for playback

2. When it is time to render, calculate number of samples I should write to the line. For example, if my last game frame took 10ms, and I'm using a 44010hz playback format, I should be writing 4401 samples this frame. But won't there be problems in the playback if the next game frame took even a little longer (buffer underflow), or shorter, than the supposed 10ms?

3. Get 4401 samples from all the audio data streams, mix them all together (basically just adding amplitudes of each sample correct?), and then write that to the SourceDataLine (ensuring that the SourceDataLine has a large enough buffer so that it doesn't block).

May I have a look at the code for your implementation? That would help to confirm that I'm going about it the right way and reduce any (potential) headaches tremendously.

Thanks again.

[erikd]

(basically just adding amplitudes of each sample correct?)

Yes, but don't forget to divide the amplitudes by the number of channels too, to prevent clipping. Best to do this *after* adding all channels, not before. You will get noise if you divide the amplitudes before mixing.
But maybe you won't need to mix everthing yourself. In my case, I emulate sound chips, and I use one sourcedataline per chip, but mix the channels of each chip myself. For a game I guess it's another story. But I still think you shouldn't have separate threads for each SourceDataLine, which was my main point.

But won't there be problems in the playback if the next game frame took even a little longer (buffer underflow), or shorter, than the supposed 10ms?

I guess you could do some buffering to prevent this, although I do believe there is already some buffering before the data actually goes to your sound board.
But if you do need to implement buffering, maybe you could set some target amount of latency based on how much you expect the framerate to vary. So the first frame you could fill the buffer until the target latency would be reached, then you fill the buffer with the number of samples needed for each frame (like we discussed). If the next frame will take longer, you won't get buffer underruns except if that frame will take longer than the set amount of latency. (I don't know if this is the best implementation, just thinking out loud here)
In my code I simply rely on the game (well, actually an emulator) to run exactly the correct amount of FPS which of course makes things way more easy, which is why my source won't be all that helpful to you.
But like I said, I think there's already some buffering going on (inside javax.sound or maybe in the sound driver, I'm not sure), so maybe this extra buffering isn't even necessary (I never experienced any buffer underruns in my emulator at all yet, which doesn't implement buffering at all).

Arrgh! Grappling with javax.sound to make it work properly is needlessly tedious indeed...

Heh, I actually find javax.sound a pretty convenient and straightforward API :-) I think you might have had this kind of problems with any other sound API as well as I believe the multiple threads are the cause of your problems.