Writing my own software audio mixer

[paulscode]

Hello, all.  My next big project is to write my own software audio mixer, so that I can play multiple audio inputs on a single SourceDataLine.  I need to be able to do the following things with PCM audio data:

1) Mixing multiple inputs into one 2-channel (left/right) output, maintaining synchronization between the channels
2) Converting between (or mixing directly) various audio formats (different sample rates/sample sizes/channels)
3) Changing frequency (pitch)
4) Changing gain (volume) independently on each channel

I want to be able to do this in pure Java, completely programatically (i.e. no linking with 3rd-party libraries).  I'm looking for all tips, source code, tutorials, that kind of thing.  I am proficient in several programming languages, so references don't necessarily have to be written specifically for Java.

[ShannonSmith]

All fairly trivial except for:

1) Low latency (tricky in Java but possible).
2) Converting between sample rates and changing frequency. Both of those are not trivial to get high quality results. There are a few pretty good sites for audio stuff, musicdsp.org has some good code examples also the dsp forum at  KVR is a good place to ask questions. Columbia uni has a very good mailing list on audio dsp that I tend to search with google
eg:
sample rate conversion site:music.columbia.edu

[nsigma]

Hello, all.  My next big project is to write my own software audio mixer, so that I can play multiple audio inputs on a single SourceDataLine.  I need to be able to do the following things with PCM audio data:

1) Mixing multiple inputs into one 2-channel (left/right) output, maintaining synchronization between the channels
2) Converting between (or mixing directly) various audio formats (different sample rates/sample sizes/channels)
3) Changing frequency (pitch)
4) Changing gain (volume) independently on each channel

I want to be able to do this in pure Java, completely programatically (i.e. no linking with 3rd-party libraries).  I'm looking for all tips, source code, tutorials, that kind of thing.  I am proficient in several programming languages, so references don't necessarily have to be written specifically for Java.

From my experience, I'd suggest using small, constant size floating-point buffers throughout your audio pipeline.  This is the method used by all the Java audio libraries I know of, including my own, and will make a lot of the mixing stuff a lot easier and reduce any problems with different sample sizes.  Sample rate conversion (and changing frequency) require interpolation between a source table and your destination buffer - I'd recommend going with some sort of pluggable interpolation strategy and starting with a basic linear interpolator - quite nasty but possibly good enough, and you can then plug in better alternatives later.

For mixing (and gain) and any other sort of control change, I'd also recommend linear interpolating values through your buffer period as it's easy to get nasty clicks otherwise.

For research, you might want to look at Beads http://www.beadsproject.net, Toot http://toot.org.uk/, JASS http://www.cs.ubc.ca/~kvdoel/jass/, and possibly RasmusDSP (by the guy who wrote Gervill) http://rasmusdsp.cvs.sourceforge.net/.  Only RasmusDSP has a licence open enough for you if I recall correctly, and it's incredibly complicated and all undocumented - not that I'm one to criticise! 

I am planning on opening up some of the audio libraries under Praxis http://code.google.com/p/praxis as GPL w/CPE in the near future, but I know you don't like that licence!    Feel free to look around the repo though - audio.ops, audio.servers and rapl are the 3 things to look under.  Just be warned it's a little untidy in there, and there may be dragons! 

All fairly trivial except for:
1) Low latency (tricky in Java but possible).

For low latency, worth having a look at this page http://frinika.wikispaces.com/JavaSoundAudioServer  I'm using a similar principle in Praxis, it's not that tricky, and it gives good results.  In fact, Frinika is worth a look for what's possible with Java audio.

Hope that's vaguely useful, and good luck with it.

Best wishes, Neil

Games published by our own members! Check 'em out!

[Jono]

One approach to doing the pitch shifting is to use a phase vocoder. With this you can slow down/speed up a sound without changing the pitch. By resampling up or down respectively, you can then change the longer or shorter sounds into ones with the original duration but different pitch.

The quality of this might not be as good as you need, but since I had most of this lying around already I thought I'd put it together anyway. So here's a basic framework for a phase vocoder in Java. The code's all mine, and you're welcome to take any bits you want and chuck a BSD license or whatever on them.

Here is an example of an original sound, the same sound shortened, and the same sound extended.

Non-periodic sounds (noise) is where the phase vocoder tends to fail. Also, to handle low frequency sounds you will need to use a larger window size, which increases the processing overhead.

[paulscode]

Awesome, thank you very much for the code samples!  This will be very helpful.