Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (475)
Games in Android Showcase (106)
games submitted by our members
Games in WIP (529)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  GZip stream oddness  (Read 1017 times)
0 Members and 1 Guest are viewing this topic.
Offline ryanm

Senior Member


Projects: 1
Exp: 15 years


Used to be bleb


« Posted 2009-03-07 21:43:41 »

Howdy

I've been unsing the GZIP input and output streams and have noticed that there's occasionally some bytes remaining in the underlying stream after the desired data has been read. For example:

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
byte[] data = /*some bytes generated by my code*/;

System.out.println( data.length + " source bytes" );

ByteArrayOutputStream bo = new ByteArrayOutputStream();
GZIPOutputStream gzo = new GZIPOutputStream( bo );

gzo.write( data );

gzo.flush();
gzo.close();

byte[] zipped = bo.toByteArray();

ByteArrayInputStream bi = new ByteArrayInputStream( zipped );
GZIPInputStream gzi = new GZIPInputStream( bi );

byte[] unzipped = new byte[ data.length ];

int read = 0;
int count = 0;
do
{
   count += read;
   read = gzi.read( unzipped, count, unzipped.length - count );
}
while( read > 0 );

System.out.println( "unzipped equals source data? " + Arrays.equals( data, unzipped ) );
System.out.print( bi.available() + " bytes left in bi [ " );

while( bi.available() > 0 )
{
   System.out.print( ( byte ) bi.read() + ", " );
}
System.out.println( " ]" );


give the output
1  
2  
3  
50293 source bytes
unzipped equals source data? true
6 bytes left in bi [ 36, -76, 117, -60, 0, 0,  ]


This behaviour is triggered by the input data (it's reproducible but rare) and I think should be regarded as a bug - intermittently leaving unexpected bytes in the stream is very naughty.

So, am I doing something wrong, or is this behaviour documented somewhere?
Online Riven
« League of Dukes »

JGO Overlord


Medals: 741
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #1 - Posted 2009-03-07 22:50:36 »

I bet the last call to read() in this case has a 'len' argument of 0. And read() never returns -1. Try to change the while condition, because read() is allowed to return 0 if len is 0.

I don't know the spec of the GZ format, but maybe it allows padding data?

It is odd behaviour, but you're not reading the full stream, so anything may be left inthere, even if it wouldn't yield any more bytes.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline ryanm

Senior Member


Projects: 1
Exp: 15 years


Used to be bleb


« Reply #2 - Posted 2009-03-07 23:44:45 »

Yep, that was it. So, the code to fully read the stream now looks like

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
int read = 0;
int count = 0;
do
{
   count += read;
   read = gzi.read( unzipped, count, unzipped.length - count );
}
while( count < unzipped.length );

// we've got our data, now we have to finish the gzip stream
while( gzi.available() != 0 )
{
   gzi.read();
}


Kind of ugly, and means you have to treat gzip streams differently from others. Is there a better way? Reading a byte at a time would work, but I imagine it could be inefficient on some stream types.
Enforcing that len always be 1 or more results in an index out of bounds, and read never gets to -1 in the first loop.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Riven
« League of Dukes »

JGO Overlord


Medals: 741
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #3 - Posted 2009-03-08 00:17:18 »

Hm.. I don't see a problem..?


Just read your data into a ByteBufferOutputStream(expectedSize), until EOF.

Call baos.toByteArray().


done.



1  
2  
3  
4  
5  
6  
public static void transfer(InputStream in, OutputStream out) { ... };

GZipInputStream in = ...;
ByteArrayOutputStream out = new ByteArrayOutputStream(expectedSize);
transfer(in, out);
byte[] ungz = out.toByteArray();


Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline oNyx

JGO Coder


Medals: 1


pixels! :x


« Reply #4 - Posted 2009-03-08 01:50:31 »

GZip: 10 byte header (magic+version+timestamp), optional extra header (original file name+whatever), deflated data, 8 byte footer (crc32+length).

You could try InflaterInputStream and DeflaterOutputStream.

弾幕 ☆ @mahonnaiseblog
Offline ryanm

Senior Member


Projects: 1
Exp: 15 years


Used to be bleb


« Reply #5 - Posted 2009-03-08 10:00:06 »

I reckon the root problem is the difference between the semantics of the InputStream and InflaterInputStream available() methods.
Input stream.available() returns the number of bytes left before blocking, while InflaterInputStream returns 1 if the end of stream has not yet been reached, even if there are no more bytes to be had.

When I ran into this I was deserialising an object graph that includes Other People's Code (read: probably broken), so I thought I'd do a check with available() and see if there were bytes left over after deserialisation was apparently complete - which would indicate that somewhere bytes were being written but not read. This worked fine with normal streams, but would throw occasional errors with GZIP streams thanks to this difference.
At any rate, it all works now and serialisation sanity is being more thoroughly checked elsewhere.

Cheers all!
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

ctomni231 (31 views)
2014-07-18 06:55:21

Zero Volt (27 views)
2014-07-17 23:47:54

danieldean (23 views)
2014-07-17 23:41:23

MustardPeter (24 views)
2014-07-16 23:30:00

Cero (39 views)
2014-07-16 00:42:17

Riven (41 views)
2014-07-14 18:02:53

OpenGLShaders (28 views)
2014-07-14 16:23:47

Riven (28 views)
2014-07-14 11:51:35

quew8 (25 views)
2014-07-13 13:57:52

SHC (61 views)
2014-07-12 17:50:04
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!