Java-Gaming.org Hi !
Featured games (90)
games approved by the League of Dukes
Games in Showcase (727)
Games in Android Showcase (217)
games submitted by our members
Games in WIP (796)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  AudioCue, starting point for concurrent Clip  (Read 554 times)
0 Members and 1 Guest are viewing this topic.
Offline philfrei
« Posted 2017-05-08 07:00:26 »

Jar file, with source and three audio samples can be downloaded from audiocue.jar.

The goal of the class AudioCue is to make a control that is relatively easy to use and has capabilities not available with the Java Clip (javax.audio.sampled.Clip).

This is just a starting point. There is a LOT that you can do with a Clip that I haven't implemented yet. The only thing I put in was the ability to set the volume of the play(), and to allow AudioCue to play concurrently. But the framework is there for additional features to be added on, such as looping, or playing back at various speeds, or altering the volume during playback. I'll try to find the time to add these capabilities in the next few weeks, but probably won't be able to do much until at least June.

My strategy for concurrent play back is to have all instances of the cue mixed together and output via a single javax.audio.sampled.SourceDataLine. It is possible to rig up multiple Java Clips of the same cue, to allow the sound to be played over itself, but each Clip opens up a new line for audio output. I thought it a good idea to keep all the occurrences grouped in the same output line. NOTE: if you are using audio files that were recorded at loudest possible volume, you will have to back off on the volume for concurrent playback to avoid distortion.

Some code reading/review would be great! I'm open to others tinkering and proposing changes or using the code and making their own changes.

The code has some diagnostics in it that can be observed if run from the console. You'll probably want to comment them out if you use this code yourself.

The "test" class file will first play a normal Java Clip, then three overlapping plays of the AudioCue. The pauses are on the long side, to accommodate the longest of the three audio samples that I included. The included samples are a gunshot, a bell, and a frog croak. You can use comments on the code to select which sound you wish to test.

This first file is the code for testing the class.
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
package jgo;

import java.io.IOException;
import java.net.URL;

import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.Clip;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.LineUnavailableException;
import javax.sound.sampled.UnsupportedAudioFileException;

public class TestAudioCue {

   public static void main(String[] args) throws LineUnavailableException,
      UnsupportedAudioFileException, IOException, InterruptedException
   {
      System.out.println("TestAudioCue() start");
      TestAudioCue test = new TestAudioCue();
      URL url;
      url = test.getClass().getResource("GunshotIndoor3.wav");
//      url = test.getClass().getResource("a3.wav");
//      url = test.getClass().getResource("frog.wav");
     
      System.out.println("Test Normal Java Clip");
       AudioInputStream ais = AudioSystem.getAudioInputStream(url);
       DataLine.Info info = new DataLine.Info(Clip.class, ais.getFormat());
       Clip clip = (Clip) AudioSystem.getLine(info);
       clip.open(ais);
       
      Thread.sleep(100);
       // The "play" of the normal Java Clip
      clip.start();
       Thread.sleep(8000);
       clip.close();
       
       System.out.println("Test new AudioCue");
      AudioCue cue = new AudioCue(url);
      cue.open();
      Thread.sleep(100);
         
      cue.play(0.8f);  // range of volume is a float, 0 to 1
      Thread.sleep(750);
      cue.play(0.8f);
      Thread.sleep(250);
      cue.play(0.8f);
     
      Thread.sleep(8000);
      cue.close();

      System.out.println("TestAudioCue() done");
   }

}


The next file is the AudioCue class. It contains two inner classes. AudioCuePlayer runs in its own thread. [The key "Audio Thread" classes are marked as such with comments. It is important to make sure nothing that blocks playback is added to this code.] When you open() or close() the AudioCue, that will handle setting up or destroying the SourceDataLine and thread that is used for playback. It is good to open() the cue a little bit in time prior to when you first play it. When the cue is open but not actively being played, it sends out zeros -- i.e., silence.

The other inner class is AudioCueCursor. This is a cursor/pointer into the sound data array. Unlike Clip, we have access to the sound data, which opens all sorts of possibilities. The AudioCueCursor currently just iterates through the array, and destroys itself when it is done. However, instead of destroying itself, we could add a flag to have it loop, or we could have it progress through the data by an increment other than 1 (and use linear interpolation to arrive at an output value). I have used both of these techniques successfully in my audio library.

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
72  
73  
74  
75  
76  
77  
78  
79  
80  
81  
82  
83  
84  
85  
86  
87  
88  
89  
90  
91  
92  
93  
94  
95  
96  
97  
98  
99  
100  
101  
102  
103  
104  
105  
106  
107  
108  
109  
110  
111  
112  
113  
114  
115  
116  
117  
118  
119  
120  
121  
122  
123  
124  
125  
126  
127  
128  
129  
130  
131  
132  
133  
134  
135  
136  
137  
138  
139  
140  
141  
142  
143  
144  
145  
146  
147  
148  
149  
150  
151  
152  
153  
154  
155  
156  
157  
158  
159  
160  
161  
162  
163  
164  
165  
166  
167  
168  
169  
170  
171  
172  
173  
174  
175  
176  
177  
178  
179  
180  
181  
182  
183  
184  
185  
186  
187  
188  
189  
190  
191  
192  
193  
194  
195  
196  
197  
198  
199  
200  
201  
202  
203  
204  
205  
206  
207  
208  
209  
210  
211  
212  
213  
214  
215  
216  
217  
218  
219  
220  
221  
222  
223  
224  
225  
226  
227  
228  
229  
230  
231  
232  
233  
234  
235  
236  
237  
238  
239  
240  
package jgo;

import java.io.IOException;
import java.net.URL;
import java.util.concurrent.CopyOnWriteArrayList;

import javax.sound.sampled.AudioFormat;
import javax.sound.sampled.AudioInputStream;
import javax.sound.sampled.AudioSystem;
import javax.sound.sampled.DataLine;
import javax.sound.sampled.Mixer;
import javax.sound.sampled.SourceDataLine;
import javax.sound.sampled.UnsupportedAudioFileException;
import javax.sound.sampled.Line.Info;
import javax.sound.sampled.LineUnavailableException;

public class AudioCue
{
   private final float[] cue;
   private final CopyOnWriteArrayList<AudioCueCursor> cursors;
   
   private final int bufferSize = 64;
   private final int outBufferSize = bufferSize * 2;
   private final float[] buffer = new float[bufferSize];
   
   private final AudioFormat audioFormat;
   private final Info info;
   
   private Mixer mixer;
   public void setMixer(Mixer mixer)
   {
      this.mixer = mixer;
   }
   
   public volatile boolean playerRunning;
   
   public AudioCue(URL url) throws UnsupportedAudioFileException,
      IOException, LineUnavailableException
   {
      cue = loadURL(url);
      cursors = new CopyOnWriteArrayList<AudioCueCursor>();
     
      audioFormat = new AudioFormat(
                  AudioFormat.Encoding.PCM_SIGNED,
                  44100, 16, 2, 4, 44100, false);
     
      info = new DataLine.Info(SourceDataLine.class,
            audioFormat);      
   }
   
   private float[] loadURL(URL url) throws UnsupportedAudioFileException,
   IOException
   {
      AudioInputStream ais = AudioSystem.getAudioInputStream(url);

      int framesCount = 0;
      if (ais.getFrameLength() > Integer.MAX_VALUE >> 1)
      {
         System.out.println(
               "WARNING: Clip is too large to entirely fit!");
         framesCount = Integer.MAX_VALUE >> 1;
      }
      else
      {
         framesCount = (int)ais.getFrameLength();
      }
     
      // stereo output, so two entries per frame
      float[] temp = new float[framesCount * 2];
      long tempCountdown = temp.length;

      int bytesRead = 0;
      int bufferIdx;
      int clipIdx = 0;
      byte[] buffer = new byte[1024];
      while((bytesRead = ais.read(buffer, 0, 1024)) != -1)
      {
         bufferIdx = 0;
         for (int i = 0, n = (bytesRead >> 1); i < n; i ++)
         {
            if ( tempCountdown-- >= 0)
            {
               temp[clipIdx++] =
                     ( buffer[bufferIdx++] & 0xff )
                     | ( buffer[bufferIdx++] << 8 ) ;
            }
         }
      }
     
      // TODO QUESTION: is it better to do following in above loop
      // rather than iterating twice?
      for (int i = 0; i < temp.length; i++)
      {
         temp[i] = temp[i] / 32767f;
      }
     
      return temp;
   }  

   public void open()
   {
      AudioCuePlayer player = new AudioCuePlayer();
      Thread t = new Thread(player);
     
      t.setPriority(Thread.MAX_PRIORITY);    
      t.start();
      playerRunning = true;
   }

   public void close()
   {
      playerRunning = false;
   }
   
   public void play(float volume)
   {
      System.out.print("AudioCue.play called: " + System.currentTimeMillis());
   
      AudioCueCursor cursor = new AudioCueCursor();
      cursors.add(cursor);
      cursor.play(volume);
      System.out.println(", AudioCueCursor.play called:"
            + System.currentTimeMillis());
      System.out.println("(play) playing Cursors:" + cursors.size());
   }  
   
   private class AudioCueCursor
   {
      volatile boolean  isPlaying;
      int idx = 0;  
      float volume = 0;
     
      // TODO: make idx a float if/when we implement
      // variable speed play back, and add LERP
      void play(float volume)
      {
         idx = 0;
         this.volume = volume;
         isPlaying = true;
      }
     
      // Audio Thread Code
      public void read(float[] buffer)
      {
         if (isPlaying)
         {
            float audioData = 0;
            for (int i = 0, n = buffer.length; i < n; i++)
            {
               audioData = idx < cue.length ? cue[idx] : 0 ;
               buffer[i] += volume * audioData;
               idx++;
            }
         }
         isPlaying = idx < cue.length;
         if (!isPlaying)
         {
            cursors.remove(this);
            System.out.println("(read/remove) playing Cursors:" + cursors.size());
         }
      }
   }
   
   private class AudioCuePlayer implements Runnable
   {
      SourceDataLine sdl;
     
      AudioCuePlayer()
      {
         System.out.println("Opening SDL");
         try {
            sdl = getSDL();
            sdl.open();
            sdl.start();
         }
         catch (LineUnavailableException e)
         {
            e.printStackTrace();
         }
      }
     
      // Audio Thread Code
      public void run()
      {
         while(playerRunning)
         {
            // Start with 0-filled buffer, send out silence
            // if nothing playing.
            for (int i = 0; i < bufferSize; i++) buffer[i] = 0;
   
            for (AudioCueCursor acc:cursors)
            {
               if (acc.isPlaying)
               {
                  // sum data into buffer
                  acc.read(buffer);
               }
            }
           
            sdl.write(convertToAudioBytes(buffer), 0, outBufferSize);
         }
     
         System.out.println("closing sdl");
         sdl.drain();
         sdl.close();
         sdl = null;
      }
   }

   // Set up Java's output line
   private SourceDataLine getSDL() throws LineUnavailableException
   {
      if (mixer == null)
      {
         // default = Java's selection
         return (SourceDataLine)AudioSystem.getLine(info);
      }
      else
      {
         // an actively chosen Line
         return (SourceDataLine) mixer.getLine(info);
      }
   }

   // Audio Thread Code
   private byte[] convertToAudioBytes(float[] buffer)
   {
      byte[] audioBytes = new byte[outBufferSize];
     
      for (int i = 0; i < bufferSize; i++)
      {
         buffer[i] *= 32767;
         
         audioBytes[i*2] = (byte) buffer[i];
         audioBytes[i*2 + 1] = (byte)((int)buffer[i] >> 8 );
      }
   
      return audioBytes;
   }
}


Some more notes (I have a reputation for wordiness to maintain), and things to add:

> Audio Data form:
I chose to go ahead and convert the audio bytes to normalized float data (ranging from -1 to 1). We could have converted the bytes to shorts and done our audio mixing in that format. Maybe people will prefer that form in order to save a multiplication and division per frame. I think for many algorithms (not implemented yet) it is just easier to work with normalized floats.

> Volume changes during playback:
Well for short cues, you can just include the desired volumes with the play() command and taper or grow as you please with the successive plays. For longer cues, the question is whether to add a volume control that affects all concurrently playing cues (a "master volume") or for just a single playback instance. For this, my plan would be to create a hook in the AudioCueCursor, and make the AudioCueCursor a return value of the play() method. Changing volumes abruptly can create discontinuities in the sound data that results in clicks. My strategy has been to take commands to alter volume and incorporate them over the course of 1028 frames, in linear increments. This brings up another consideration: the volume parameter does not have a linear effect. You will probably want to put in something like a X^3 function if you want a semblance of, for example, volume 0.25 being one half as loud as 0.5.

> Audio playback buffer:
My choice of a buffer size is kind of arbitrary. Others probably have a better grasp of what values would allow the best throughput:safest trade offs. Please edit and let me know what works best for you. To be honest, the audio library I made for myself has, basically, a buffer of 1 (i.e., no buffer, just goes forward one frame at a time). This has worked out so far and allowed me to write a functional event system with per frame accuracy. But since the goal here is just to supplement the Java audio controls of Clip and SourceDataLine, it seemed logical to go for a greater throughput afforded by using a buffer. [This is part of the reason it took so long for me to write this--having to convert to using a buffer which none of my code uses.]

> Should I have stuck with start() instead of play()? It seems to me that play(), with parameters like volume, panning (to do), the addition of a AudioCueListener (also, to do), is the clearer command name.

> We probably should have a mono cue version that can be stereo panned. Panning stereo cues doesn't make a whole lot of sense to me. The panning algorithm could either use volume or using delay.

> I like doing more than just simple looping. I like having options to overlap the edges, and have done this in my own audio library. Cues with overlaps can have a nice smooth continuity where some loops have clicks over the breaks. It is mostly a matter of setting the overlap size (in frames) and deciding on the blending function (linear, or just add together, or various curves sometimes work best). I haven't done the following yet on my own library, but I think it would be neat to have a smoothly looping sound that is subject to real time speed and volume changes.

> Once one can do overlapping of edges, it is a short distance to extracting 'macro-granules' from the sound data body and stringing them together. This strategy can be used to create endless-non-repeating sounds such as campfires or brooks.

> TIMING:
Okay, the timing of the playbacks is going to be subject to some variance. This is kind of inevitable, as Java does not offer real time guarantees. In particular, the first play may lag by a millisecond or two, relative to the following plays, as we go from compiled to hot. This can be observed in the diagnostic comments I put in the code. Also, if you do a string of plays, all at exactly 100 millis, for example, chances are there will be some wobble to the timing. Another contributor to this is that we can't predict when Java switches between threads or when it chooses to garbage collect. Java DOES do a good time of maintaining playback steadiness of existing sound--and CAN provide very solid timing, but doing so requires tapping into the audio thread and counting elapsed frames. At least, that is what I do with the event system I wrote. Even so, the timing is pretty good. The AudioClip should be on a par with the Java Clip.

music and music apps: http://adonax.com
Pages: [1]
  ignore  |  Print  
 
 

 
Archive (299 views)
2017-04-27 17:45:51

buddyBro (486 views)
2017-04-05 03:38:00

CopyableCougar4 (933 views)
2017-03-24 15:39:42

theagentd (946 views)
2017-03-24 15:32:08

Rule (955 views)
2017-03-19 12:43:22

Rule (924 views)
2017-03-19 12:42:17

Rule (925 views)
2017-03-19 12:36:21

theagentd (988 views)
2017-03-16 05:07:07

theagentd (903 views)
2017-03-15 22:37:06

theagentd (696 views)
2017-03-15 22:32:18
List of Learning Resources
by elect
2017-03-13 14:05:44

List of Learning Resources
by elect
2017-03-13 14:04:45

SF/X Libraries
by philfrei
2017-03-02 08:45:19

SF/X Libraries
by philfrei
2017-03-02 08:44:05

SF/X Libraries
by SkyAphid
2017-03-02 06:38:56

SF/X Libraries
by SkyAphid
2017-03-02 06:38:32

SF/X Libraries
by SkyAphid
2017-03-02 06:38:05

SF/X Libraries
by SkyAphid
2017-03-02 06:37:51
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!