YUNPM - a Java Media Player

[kappa]

So in this case, even if we dont use H264 files but Theora with "YUNPM", we still could not bundle it legally for free with our game, because it includes a H264 decoder.
If its possible to seperate those parts, it would work out. With VLC and GStreamer we have all these dlls and can delete them one by one, depending on license and requirement.

You can compile ffmpeg yourself to include as many or as few (even one) codecs as you like. Besides you'll probably want to do that just to reduce its size as by default it includes the whole kitchen sink.

[Cero]

So in this case, even if we dont use H264 files but Theora with "YUNPM", we still could not bundle it legally for free with our game, because it includes a H264 decoder.
If its possible to seperate those parts, it would work out. With VLC and GStreamer we have all these dlls and can delete them one by one, depending on license and requirement.

You can compile ffmpeg yourself to include as many or as few (even one) codecs as you like. Besides you'll probably want to do that just to reduce its size as by default it includes the whole kitchen sink.

ah yes, so its basically the same just that its all compiled into one big binary.

[nsigma]

WebM/VP8 are relatively young even compared to Theora and their legality is still questionable.

Old is better?  On that basis I'll stick with Java 1.1 for now then! 

Not sure if you meant 'their' as in Theora and VP8, because both are still open to the question of patent legality (bloody software patents).  Personally, I'd pick the better quality one and stop worrying about it!

You can compile ffmpeg yourself to include as many or as few (even one) codecs as you like. Besides you'll probably want to do that just to reduce its size as by default it includes the whole kitchen sink.

hmm, that's a big negative for ffmpeg in my opinion then.  I use Java so I don't have to worry about doing cross-compilation on every target platform!  That's a big win for the options with a dynamically loaded plugin architecture - delete unnecessary codecs, done!

Games published by our own members! Check 'em out!

[Riven]

Shipping licensed codecs is not stealing!



_{(and please define stealing, in the next 8 replies)}

[OttoMeier]

what about javafx for video playback?
https://blogs.oracle.com/javafx/entry/mpeg_4_multimedia_support_in

[Mads]

Is there a few select formats and codecs that'll work, and will it be easy to plug in new ones/home brew?

[Cero]

Is there a few select formats and codecs that'll work, and will it be easy to plug in new ones/home brew?

The Theora & Vorbis Combo is absolutely great and legal to use.
And sure you COULD write own codecs... like you could write your own OS  D=

[Mads]

Is there a few select formats and codecs that'll work, and will it be easy to plug in new ones/home brew?

The Theora & Vorbis Combo is absolutely great and legal to use.
And sure you COULD write own codecs... like you could write your own OS  D=

Well, who knows? It's nice to experiment, and sometimes new formats and codecs comes along. It would be nice to get support, even if Riven is no maintaining the library. This is just playback, right? No editing or processing?

[nsigma]

what about javafx for video playback?
https://blogs.oracle.com/javafx/entry/mpeg_4_multimedia_support_in

Wonder what that uses ... oh, GStreamer! 

Having said that, relying on a paid-for h264 codec on Linux - like that's going to work!  Hopefully they will support other codecs (discussion on WebM in comments) - it will when it's open-sourced that's for sure.

[princec]

A lot of those format wars actually rage in the sky far above our heads. Most codecs can compress data in such a ways as to be of very high quality, it's the bitrates that matter - and the bitrates seriously do not matter at all for games. It's all about streaming formats, because for these massive behemoths that serve 10,000,000 videos a day, conserving 10% more bandwidth is a serious, serious consideration.

For the rest of us though... the simplest, no-license-required format is all we need. Who cares if the video data is 20mb instead of 10mb.

Cas

Games published by our own members! Check 'em out!

[nsigma]

@princec - generally agree with you on all that - in fact, for my own usage I almost always use MJPEG, because that also allows for seamless skipping and easy change of playback rate, etc. (irrelevant to what most people around here want to do).  However, your statement maybe contradicts your earlier point about the quality of h264 - I mentioned VP8 because it's the most similar (theoretically) unencumbered codec.

[theagentd]

It's working. May I recommend enabling V-sync? This amount of stuttering is even too much for me.

[princec]

I did find that mjpeg took absolutely masses of CPU to decode though (not to mention that it uses an awful lot more space than MPEG). In fact I knocked up an MJPEG video player a couple of years in plain old Java inside an hour but it basically is no good space wise or performance wise - still got an eye on bandwidth, and still got other uses than just plain old cutscenes, and still looking at relatively old and underpowered systems.

Cas

[theagentd]

I am impressed! ffmpeg.exe takes barely 5% CPU (around 25% on one of 4 cores), while Java switches between 0% and 1%. The total is the same or even slightly under what Media Player Classic uses for me. GPU load is at just 6-7%.

Oh, that was for a 1080p H264 video!

[Mads]

I get one of these, under Windows.

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16

Exception in thread "main" java.lang.UnsupportedClassVersionError: net/indiespot /media/VideoPlaybackTest : Unsupported major.minor version 51.0         at java.lang.ClassLoader.defineClass1(Native Method)         at java.lang.ClassLoader.defineClassCond(Unknown Source)         at java.lang.ClassLoader.defineClass(Unknown Source)         at java.security.SecureClassLoader.defineClass(Unknown Source)         at java.net.URLClassLoader.defineClass(Unknown Source)         at java.net.URLClassLoader.access$000(Unknown Source)         at java.net.URLClassLoader$1.run(Unknown Source)         at java.security.AccessController.doPrivileged(Native Method)         at java.net.URLClassLoader.findClass(Unknown Source)         at java.lang.ClassLoader.loadClass(Unknown Source)         at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)         at java.lang.ClassLoader.loadClass(Unknown Source) Could not find the main class: net.indiespot.media.VideoPlaybackTest.  Program w ill exit.

[nsigma]

@princec - I wasn't suggesting MJPEG as the right codec to use for most cases, just agreeing with your point that the concerns of online streaming codecs are not necessarily relevant.  Where MJPEG is useful is when you need a codec that is all I-frames - animating back and forth (video scratching) through a video file could have some interesting uses in a game methinks.

@theagentd - wonder if you fancy having a look at Cero's code above and see how it compares?  I believe some of the texture writing stuff came from you (for VLCJ) originally.  Would be even better to update it to the same pixel upload method as Riven is now using.  I don't have a Windows dev box, though will have a look on Linux.

[princec]

@Mads - it's compiled for Java 7.

Cas

[nsigma]

Well, just did a test of YUNPM vs the GStreamer code Cero posted, but on Linux.  Used a longer video of same resolution for both.

0.7.3 - 4x the CPU usage of GStreamer - as expected given the extra encoding / decoding.
0.7.5 - identical performance.
0.7.7 - YUNPM slightly ahead.

I made one change to the code Cero posted - an obvious optimization to only update the texture when there's a new frame.  Other than that there's a range of optimizations that could be made (eg. the conversion to RGB32 is unnecessary) as well as the optimizations that went into YUNPM 0.7.7.  Should try without Slick too, not sure what overhead that might be having - this was just a quick test!  Given the bulk of the CPU time is in the codec, and they're both using the same one, then not really a surprise that they're now comparable - answering my own earlier question, in-process and out-of-process doesn't seem to make a lot of difference (whereas the MJPEG encoding definitely did!).

I'll tidy up the GStreamer bundling code at some point soon.  Main reason to use it would be if you want the added features of GStreamer I guess (position, speed controls; online streaming; webcam input, etc. etc.).

[theagentd]

@theagentd - wonder if you fancy having a look at Cero's code above and see how it compares?  I believe some of the texture writing stuff came from you (for VLCJ) originally.  Would be even better to update it to the same pixel upload method as Riven is now using.  I don't have a Windows dev box, though will have a look on Linux.

Compare what to what?

[nsigma]

Compare what to what?

As per my last post, but on Windows.  I was under the impression that the code Cero had to work with VLC, and I hacked around to function with GStreamer for him, was originally written by you?

[Cero]

Should try without Slick too

I would love to, just for simplicity/purity sake. But we use TextureImpl.getLastBind() and I dont speak OpenGL well enough to replace it =D

[theagentd]

Compare what to what?

As per my last post, but on Windows.  I was under the impression that the code Cero had to work with VLC, and I hacked around to function with GStreamer for him, was originally written by you?

I just wrote the code to display the ByteBuffer that JVLC gave him, and it did not use PBOs, so it definitely won't be faster on the Java side. I can't speak for ffmpeg vs VLC though, but I don't think there's too much of a difference there...

[nsigma]

Shipping licensed codecs is not stealing!

No, though possibly problematic.  Just had a thought related to that - if your build of FFMPEG can play h264 isn't it GPL?  (Mind you, the GStreamer libs Cero linked to also include the same GPL code, but this can be pulled out by deleting the plugin).

@theagentd - Never mind, I meant a comparison of Riven's code with FFMPEG and Cero's with GStreamer - had a go with that on Linux as mentioned, but was curious about Windows.  Basically, what if any overhead is created by doing this in separate processes vs one (the codec code in GStreamer and VLC is from FFMPEG anyway).  Not a lot by the look of it!

[Cero]

Shipping licensed codecs is not stealing!

No, though possibly problematic.  Just had a thought related to that - if your build of FFMPEG can play h264 isn't it GPL?  (Mind you, the GStreamer libs Cero linked to also include the same GPL code, but this can be pulled out by deleting the plugin).

I deleted much in there, hopefully as much as possible for just playing theora&vorbis, certainly removed all codecs that sounded like licensed codecs I know

[Riven]

Version 0.7.8 of YUNPM

• Decoupled playback framerate from video framerate
  
  • Allows rendering 24fps movie in game running at 60fps
  • Allows rendering game while movie is paused
  • To showcase this feature, an annoying translucent butter smooth rotating quad is rendered on top of the video
  • If you feel uncomfortable with the licensing of specific codecs in ffmpeg, please build ffmpeg yourself without said codecs
• Discarding audio stream from video, by passing null AudioRenderer
• Support for Fullscreen and v-sync
• Jar released for Java6 again (sorry for the previous screwup)

Download files

• http://indiespot.net/files/projects/medialib/

Notes

• with the decoupling of movie playback and framerate, PBO performance has taken a nose dive. Currently it's only slightly faster than directly updating textures, which means the CPU usage is significantly higher than in v0.7.7. As I'm new to the asynchronous behaviour of PBO's, I'd appreciate it if somebody could take a look as to why this happened. In the meantime I'll try to fix it myself, obviously.

Hilarious sample code

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47

class VideoPlaybackTest {    public static void main(String[] args) throws Exception {       File movieFile = new File(args[0]);       boolean audioEnabled = true;       VideoRenderer videoRenderer = new OpenGLVideoRenderer(movieFile.getName());       AudioRenderer audioRenderer = audioEnabled ? new OpenALAudioRenderer() : null;       if (videoRenderer instanceof OpenGLVideoRenderer) {          OpenGLVideoRenderer opengl = (OpenGLVideoRenderer) videoRenderer;          opengl.setFullscreen(false); // uses current desktop resolution         opengl.setVSync(true);          opengl.setRenderRotatingQuad(true);       }       VideoPlayback playback = new FFmpegVideoPlayback(movieFile);       playback.setCoupleFramerateToVideo(false);       playback.startVideo(videoRenderer, audioRenderer);       /**        * oldskool controls!!        */       BufferedReader br = new BufferedReader(          new InputStreamReader(new BufferedInputStream(System.in)));       while (true) {          String line = br.readLine();          if (line.equals("mute")) {             playback.setVolume(0.0f);          } else if (line.equals("half")) {             playback.setVolume(0.5f);          } else if (line.equals("full")) {             playback.setVolume(1.0f);          } else if (line.equals("pause")) {             playback.pause();          } else if (line.equals("resume")) {             playback.resume();          } else {             System.out.println("wait what?");          }       }    } }

[Riven]

It seems that PBO performance is related to vsync and decoupling video framerate from render framerate:

Test 1:	vsync	coupled	texture update takes 1.6ms	25fps (video framerate)	Java CPU usage: 7%
Test 2:	vsync	decoupled	texture update takes 17.2ms	60fps (vsync)	Java CPU usage: 17%
Test 3:	no vsync	coupled	texture update takes 1.6ms	25fps (video framerate)	Java CPU usage: 7%
Test 4:	no vsync	decoupled	texture update takes 5.3ms	180fps	Java CPU usage: 28%

(CPU usage as reported by the task-manager on my quadcore, so multiply by four for usage on 1 core)


Some code for those interested:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32

private void updateTexture(ByteBuffer rgb) {       if (rgb.remaining() != videoWidth * videoHeight * 3) {          throw new IllegalStateException(rgb.remaining() + " <> " + (videoWidth * videoHeight * 3));       }       // snippet of relevant PBO code      {          glBindBuffer(GL_PIXEL_UNPACK_BUFFER, pboHandle);          pboBuffer = glMapBuffer(GL_PIXEL_UNPACK_BUFFER, GL_WRITE_ONLY, pboBuffer);          //System.out.println(NativeHacks.getBufferAddress(pboBuffer));         pboBuffer.put(rgb);          pboBuffer.flip();          rgb.flip();          glUnmapBuffer(GL_PIXEL_UNPACK_BUFFER);          glBindTexture(GL_TEXTURE_2D, textures[textureIndex & 1]);          glTexSubImage2D(GL_TEXTURE_2D, 0/* level */, 0, 0, videoWidth, videoHeight, GL_RGB, GL_UNSIGNED_BYTE, 0);          glBindBuffer(GL_PIXEL_UNPACK_BUFFER, 0);       }       // bind the texture, that has been filled in the previous frame (if any)      if (textureIndex++ == 0) {          currentFrameTexture = textures[0];       } else {          currentFrameTexture = textures[textureIndex & 1];       }    }

[Spasi]

I'm getting horrible stuttering with the latest version, not only higher CPU usage. Texture updates take between 0.6ms and 50ms, very unstable. First I tried to not clear the videoUpdateBuffer after each update and that fixed it. The problem is most likely related to the GL_STREAM_DRAW flag; we're telling OpenGL that we'll be streaming continuously but end up doing it every X frames. That probably hits a bad path in the driver.

The proper fix was discarding the old FBO data prior to mapping. Add this:

1	glBufferData(GL_PIXEL_UNPACK_BUFFER, videoWidth * videoHeight * 3, GL_STREAM_DRAW);

before the call to glMapBuffer and it will be fine.

I also tried CPU asynchronous updates, as described here. That's a decent win on CPU usage at the cost of double the memory usage (you need 2 ping-pong PBOs).

edit: The ping-pong version. Probably needs to be made optional.

[Riven]

I incorporated your changes, but I don't see any improvement. I didn't have the stuttering, so that might very well be fixed, therefore I released v0.7.9.

I even tried triple buffering (3 textures, 3 PBOs) but that gave me exactly the same results.

[Spasi]

Well, the thing with vsync is that its implementation differs considerably between vendors and architectures. On my AMD for example I'm pretty sure it just busy-spins a lot, hence the higher CPU usage. Also, what you're measuring as the texture update may very well include the SwapBuffers time. All GL calls are asynchronous, the corresponding action may happen long after the call returns, including SwapBuffers. The call to glMapBuffer happens to be the first call that requires some kind of pipeline flush, so you end up "paying" the vsync cost at that time. That doesn't make PBOs slower, it simply means that timing on the CPU-side is inaccurate by nature.

Add a timer around Display.update and you'll see that it takes less than 1ms when the PBO update takes 16+ms (16.6 ms = 60fps vsynced).

[Riven]

That makes sense.

16.6ms(60Hz) + 1.5ms(coupled) = 18.1ms
That 1.5ms is probably partly covered by v-sync wait/busyloop, resulting in 17.2ms per frame.

I'd expected that modern drivers are better at syncing than a busyloop, but you could argue it's the fastest way, and nothing else matters.