Java-Gaming.org Hi !
Featured games (91)
games approved by the League of Dukes
Games in Showcase (808)
Games in Android Showcase (239)
games submitted by our members
Games in WIP (872)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  YUNPM get frame is black  (Read 9847 times)
0 Members and 1 Guest are viewing this topic.
Offline JESTERRRRRR
« Posted 2015-06-28 17:07:35 »

Hi, I am trying to use YUNPM to get frames of video onto a BufferedImage. My method works for the first 3 frames but then stops returning a bytebuffer, just null. Here is what I am doing:
1  
2  
File film = new File(path.getAbsolutePath()+"\\Assets\\Movies\\1.mp4");//This part works as it plays the sound from the sample video
player = new MoviePlayer(film);

Then to get a frame
1  
2  
player.tick();
ByteBuffer b = player.movie.videoStream().pollFrameData();

After the first 3 times it returns null. It is displaying correctly at least one frame, but I'm not sure the 2nd and 3rd are actually different from the first it displays. I have also tried the following after the above
1  
2  
player.movie.videoStream().freeFrameData(b);
player.movie.onUpdatedVideoFrame();

And while this does let me get more than 3 frames, they are bizzare colours and all over the place, with it eventually giving me
1  
java.lang.IllegalArgumentException: Number of remaining buffer elements is 528384, must be at least 921600.

Has anyone else used this and knows what I am doing wrong?
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #1 - Posted 2015-06-28 17:44:25 »

There is no need to bring in *movie* playback, which syncs with audio. As you're not signalling you played audio samples, the video stream will eventually stall. I'll give you some working code in 10 minutes. Gotta finish some cookin' first.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2015-06-28 18:20:10 »

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
import java.awt.image.BufferedImage;
import java.awt.image.DataBufferByte;
import java.io.DataInputStream;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import net.indiespot.media.impl.FFmpeg;
import net.indiespot.media.impl.VideoMetadata;

public class TestFrameExtraction {
   public static void main(String[] args) throws Exception {
      File movieFile = new File(args[0]);

      VideoMetadata meta = FFmpeg.extractMetadata(movieFile);
      InputStream rgb24Stream = FFmpeg.extractVideoAsRGB24(movieFile, 0);

      DataInputStream videoStream = new DataInputStream(rgb24Stream);

      BufferedImage buf = new BufferedImage(meta.width, meta.height, BufferedImage.TYPE_3BYTE_BGR);
      byte[] rgb = accessRasterByteArray(buf);
      for (int frame = 0; true; frame++) {
         readFrame(videoStream, rgb);
         swapChannels(rgb);
         System.out.println("frame: " + frame + " (" + meta.width + "x" + meta.height + ")");
      }
   }

   private static void readFrame(DataInputStream videoStream, byte[] dst) throws IOException {
      // videoStream.readFully(dst);
      int chunk = 64 * 1024;
      for (int off = 0; off != dst.length;) {
         int len = Math.min(chunk, dst.length - off);
         videoStream.readFully(dst, off, len);
         off += len;
      }
   }

   private static byte[] accessRasterByteArray(BufferedImage src) {
      return ((DataBufferByte) src.getRaster().getDataBuffer()).getData();
   }

   private static void swapChannels(byte[] rgb) {
      // BGR <-> RGB
      for(int i=0; i<rgb.length; i+=3) {
         byte tmp = rgb[i+0];
         rgb[i+0] = rgb[i+2];
         rgb[i+2] = tmp;
      }
   }
}

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline JESTERRRRRR
« Reply #3 - Posted 2015-06-29 04:23:03 »

Thank you! This is almost perfect, I'm having 1 tiny problem though..



I am getting the colors on the left, instead of how it should be on the right. I'm assuming this is since its BGR. Do I needs to convert the BufferedImage, or is there a quicker way I can get it as RGB?
Either way, this is brilliant thanks.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #4 - Posted 2015-06-29 08:36:21 »

BufferedImage sadly doesn't have a 3BYTE_RGB type Roll Eyes

1  
2  
3  
4  
5  
6  
// BGR <-> RGB
for(int i=0; i<rgb.length; i+=3) {
   byte tmp = rgb[i+0];
   rgb[i+0] = rgb[i+2];
   rgb[i+2] = tmp;
}


Keep in mind YUNPM uses STDIN/STDOUT to 'pipe' raw video/audio data from a separate FFMPEG process. This means that for 1920x1080 @ 24Hz, over 140MB/sec is transfered over I/O channels not really meant for this bandwidth (on Windows, at least). The overhead is generally acceptable, but with Full HD and higher resolutions and/or framerates, you need to look for alternatives.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline nsigma
« Reply #5 - Posted 2015-06-29 13:59:39 »

BufferedImage sadly doesn't have a 3BYTE_RGB type Roll Eyes

hmmm .. possible to make one though (code not tested - based on this)

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
WritableRaster raster = Raster.createInterleavedRaster(
        DataBuffer.TYPE_BYTE, width, height,
        width * 3, // scanlineStride
        3, // pixelStride
        new int[]{0, 1, 2}, // bandOffsets
        null);

ColorModel colorModel = new ComponentColorModel(
        ColorSpace.getInstance(ColorSpace.CS_sRGB),
        new int[]{8, 8, 8}, // bits
        false, // hasAlpha
        false, // isPreMultiplied
        ComponentColorModel.OPAQUE,
        DataBuffer.TYPE_BYTE);

BufferedImage img = new BufferedImage(colorModel, raster, false, null);

Praxis LIVE - hybrid visual IDE for (live) creative coding
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #6 - Posted 2015-06-29 15:04:15 »

I think that triggers an extraordinarily slow path when rendering. persecutioncomplex

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline nsigma
« Reply #7 - Posted 2015-06-29 15:15:47 »

I think that triggers an extraordinarily slow path when rendering. persecutioncomplex

Quite possibly, although at worst it's probably* only doing the same byte rearranging you are.  Wink

* probably implies logic in Java2D  persecutioncomplex

Praxis LIVE - hybrid visual IDE for (live) creative coding
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #8 - Posted 2015-06-29 15:35:34 »

A decade ago I looked into the Java2D (software) backend, and for any non-standard type, it basically did pretty 'impressive' per pixel operations. It's like doing bufimg.setRGB(x, y, value) for every pixel. That's a fancy trip down the rabbit hole, btw. Back in the day every pixel op involved a few object creations and dozen layers of indirection. I doubt much has changed, especially considering that people report factor 1000 slowdowns for some paths in the link you mentioned.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline nsigma
« Reply #9 - Posted 2015-06-29 16:37:56 »

That's a fancy trip down the rabbit hole, btw.

Been there, done that.  IIRC though it should in this case pass everything to the native conversion loops in one go rather than per pixel, hence blindly hoping that the OP might find it fast enough.  It's a quicker thing to test.

Does YUNPM give any way to specify what color model FFMPEG provides the data?  Easy to modify to BGR?

especially considering that people report factor 1000 slowdowns for some paths in the link you mentioned.

That also mentions some versions of OSX, that makes me think it might refer to the old Quartz renderer - nice and fast in some circumstances, FUBAR in others.

Praxis LIVE - hybrid visual IDE for (live) creative coding
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Riven
Administrator

« JGO Overlord »


Medals: 1371
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #10 - Posted 2015-06-29 16:49:15 »

You could add a method that takes the format as a parameter.
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
package net.indiespot.media.impl;

...

public class FFmpeg
{
   ...

   public static InputStream extractVideo(File srcMovieFile, int seconds, String pixfmt) throws IOException {
      return streamData(new ProcessBuilder().command(//
         FFMPEG_PATH, //
         "-ss", String.valueOf(seconds), //
         "-i", srcMovieFile.getAbsolutePath(), //        
         "-f", "rawvideo", //
-         "-pix_fmt", "rgb24", //
+         "-pix_fmt", pixfmt, //
         "-" //
      ));
   }

   ...
}


List of all supported formats:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
72  
73  
74  
75  
76  
77  
78  
79  
80  
81  
82  
83  
84  
85  
86  
87  
88  
89  
90  
91  
92  
93  
94  
95  
96  
97  
98  
99  
100  
101  
102  
103  
104  
105  
106  
107  
108  
109  
110  
111  
112  
113  
114  
115  
116  
117  
FLAGS NAME            NB_COMPONENTS BITS_PER_PIXEL
-----
IO... yuv420p                3            12
IO... yuyv422                3            16
IO... rgb24                  3            24
IO... bgr24                  3            24 <--- there we go
IO... yuv422p                3            16
IO... yuv444p                3            24
IO... yuv410p                3             9
IO... yuv411p                3            12
IO... gray                   1             8
IO..B monow                  1             1
IO..B monob                  1             1
I..P. pal8                   1             8
IO... yuvj420p               3            12
IO... yuvj422p               3            16
IO... yuvj444p               3            24
..H.. xvmcmc                 0             0
..H.. xvmcidct               0             0
IO... uyvy422                3            16
..... uyyvyy411              3            12
IO... bgr8                   3             8
.O..B bgr4                   3             4
IO... bgr4_byte              3             4
IO... rgb8                   3             8
.O..B rgb4                   3             4
IO... rgb4_byte              3             4
IO... nv12                   3            12
IO... nv21                   3            12
IO... argb                   4            32
IO... rgba                   4            32
IO... abgr                   4            32
IO... bgra                   4            32
IO... gray16be               1            16
IO... gray16le               1            16
IO... yuv440p                3            16
IO... yuvj440p               3            16
IO... yuva420p               4            20
..H.. vdpau_h264             0             0
..H.. vdpau_mpeg1            0             0
..H.. vdpau_mpeg2            0             0
..H.. vdpau_wmv3             0             0
..H.. vdpau_vc1              0             0
IO... rgb48be                3            48
IO... rgb48le                3            48
IO... rgb565be               3            16
IO... rgb565le               3            16
IO... rgb555be               3            15
IO... rgb555le               3            15
IO... bgr565be               3            16
IO... bgr565le               3            16
IO... bgr555be               3            15
IO... bgr555le               3            15
..H.. vaapi_moco             0             0
..H.. vaapi_idct             0             0
..H.. vaapi_vld              0             0
IO... yuv420p16le            3            24
IO... yuv420p16be            3            24
IO... yuv422p16le            3            32
IO... yuv422p16be            3            32
IO... yuv444p16le            3            48
IO... yuv444p16be            3            48
..H.. vdpau_mpeg4            0             0
..H.. dxva2_vld              0             0
IO... rgb444le               3            12
IO... rgb444be               3            12
IO... bgr444le               3            12
IO... bgr444be               3            12
I.... gray8a                 2            16
IO... bgr48be                3            48
IO... bgr48le                3            48
IO... yuv420p9be             3            13
IO... yuv420p9le             3            13
IO... yuv420p10be            3            15
IO... yuv420p10le            3            15
IO... yuv422p10be            3            20
IO... yuv422p10le            3            20
IO... yuv444p9be             3            27
IO... yuv444p9le             3            27
IO... yuv444p10be            3            30
IO... yuv444p10le            3            30
IO... yuv422p9be             3            18
IO... yuv422p9le             3            18
..H.. vda_vld                0             0
I.... gbrp                   3            24
I.... gbrp9be                3            27
I.... gbrp9le                3            27
I.... gbrp10be               3            30
I.... gbrp10le               3            30
I.... gbrp16be               3            48
I.... gbrp16le               3            48
I.... rgba64be               4            64
I.... rgba64le               4            64
..... bgra64be               4            64
..... bgra64le               4            64
IO... 0rgb                   3            24
IO... rgb0                   3            24
IO... 0bgr                   3            24
IO... bgr0                   3            24
IO... yuva444p               4            32
IO... yuva422p               4            24
IO... yuv420p12be            3            18
IO... yuv420p12le            3            18
IO... yuv420p14be            3            21
IO... yuv420p14le            3            21
IO... yuv422p12be            3            24
IO... yuv422p12le            3            24
IO... yuv422p14be            3            28
IO... yuv422p14le            3            28
IO... yuv444p12be            3            36
IO... yuv444p12le            3            36
IO... yuv444p14be            3            42
IO... yuv444p14le            3            42
I.... gbrp12be               3            36
I.... gbrp12le               3            36
I.... gbrp14be               3            42
I.... gbrp14le               3            42

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline JESTERRRRRR
« Reply #11 - Posted 2015-06-29 20:04:41 »

Now solved, thanks for the posts guys.
Pages: [1]
  ignore  |  Print  
 
 

 
Riven (847 views)
2019-09-04 15:33:17

hadezbladez (5791 views)
2018-11-16 13:46:03

hadezbladez (2603 views)
2018-11-16 13:41:33

hadezbladez (6207 views)
2018-11-16 13:35:35

hadezbladez (1499 views)
2018-11-16 13:32:03

EgonOlsen (4734 views)
2018-06-10 19:43:48

EgonOlsen (5792 views)
2018-06-10 19:43:44

EgonOlsen (3276 views)
2018-06-10 19:43:20

DesertCoockie (4175 views)
2018-05-13 18:23:11

nelsongames (5501 views)
2018-04-24 18:15:36
A NON-ideal modular configuration for Eclipse with JavaFX
by philfrei
2019-12-19 19:35:12

Java Gaming Resources
by philfrei
2019-05-14 16:15:13

Deployment and Packaging
by philfrei
2019-05-08 15:15:36

Deployment and Packaging
by philfrei
2019-05-08 15:13:34

Deployment and Packaging
by philfrei
2019-02-17 20:25:53

Deployment and Packaging
by mudlee
2018-08-22 18:09:50

Java Gaming Resources
by gouessej
2018-08-22 08:19:41

Deployment and Packaging
by gouessej
2018-08-22 08:04:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!