Java-Gaming.org    
Featured games (81)
games approved by the League of Dukes
Games in Showcase (499)
Games in Android Showcase (118)
games submitted by our members
Games in WIP (567)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  ... so never use UTF-8 encoding for binary stuff  (Read 1591 times)
0 Members and 1 Guest are viewing this topic.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 801
Projects: 4
Exp: 16 years


Hand over your head.


« Posted 2009-04-20 16:56:38 »

I rather often abuse UTF8 to encode binary to pass it into a textbased API.

Today, after years (!!), was the first time I got caught by non-reversible UTF8 encodings.

1  
2  
3  
4  
5  
         byte[] original = ....;
         String encoded = new String(original, "UTF-8");
         byte[] decoded = encoded.getBytes("UTF-8");

         Arrays.equals(original, decoded); // false!


Gotta rewrite some stuff... shame on me !

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline ryanm

Senior Member


Projects: 1
Exp: 15 years


Used to be bleb


« Reply #1 - Posted 2009-04-20 18:03:12 »

Don't know why this stuff isn't already in the JRE, but Base64 encoding works for me when I'm ramming binary data into java.util.prefs.
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 801
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #2 - Posted 2009-04-20 19:44:35 »

It is there, in rt.jar, but not supported:

sun.misc.BASE64Encoder
sun.misc.BASE64Decoder

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Abuse

JGO Knight


Medals: 12


falling into the abyss of reality


« Reply #3 - Posted 2009-04-20 21:24:01 »

I rather often abuse UTF8 to encode binary to pass it into a textbased API.

Today, after years (!!), was the first time I got caught by non-reversible UTF8 encodings.

1  
2  
3  
4  
5  
         byte[] original = ....;
         String encoded = new String(original, "UTF-8");
         byte[] decoded = encoded.getBytes("UTF-8");

         Arrays.equals(original, decoded); // false!


Gotta rewrite some stuff... shame on me !

Presumably the cause of your problem is that 'byte[] original' contains a string encoded using modified UTF-8, rather than UTF-8? (caused by inproper use of dos.writeUTF elsewhere in your app.)

Though if that's the case i'm surprised you hadn't encountered a problem sooner; it's unusual for binary data to contain no zeros!
Though perhaps the UTF-8 decoder used by the String constructor is silently accepting an Overlong encoding for zero, and you've only been caught out now because you're data contains one of the UTF-16 surrogate pair byte values. (which are also encoded overlong in modified UTF-8)

If that's the case the UTF-8 decoder used by Java is being very naughty - as accepting overlong encodings would mean it fails to meet the current Unicode compliancy requirements!

Make Elite IV:Dangerous happen! Pledge your backing at KICKSTARTER here! https://dl.dropbox.com/u/54785909/EliteIVsmaller.png
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 801
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #4 - Posted 2009-04-20 22:00:36 »

I always was 'serializing' more or less textual data, but binary in the end - like what you get from DataOutputStream when your protocol is mainly string-based.

Today it simply went bezerk, due to the need to write binary in a text SQL column: ObjectOutputStream -> utf8 -> ObjectInputStream.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline pjt33
« Reply #5 - Posted 2009-04-21 08:02:32 »

I rather often abuse UTF8 to encode binary to pass it into a textbased API.
Why not use ISO-8859-1? That has 256 characters, so it's a lot more suitable.
Pages: [1]
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Pippogeek (38 views)
2014-09-24 16:13:29

Pippogeek (29 views)
2014-09-24 16:12:22

Pippogeek (18 views)
2014-09-24 16:12:06

Grunnt (42 views)
2014-09-23 14:38:19

radar3301 (24 views)
2014-09-21 23:33:17

BurntPizza (61 views)
2014-09-21 02:42:18

BurntPizza (31 views)
2014-09-21 01:30:30

moogie (36 views)
2014-09-21 00:26:15

UprightPath (49 views)
2014-09-20 20:14:06

BurntPizza (53 views)
2014-09-19 03:14:18
List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06

List of Learning Resources
by SilverTiger
2014-07-31 11:54:12

HotSpot Options
by dleskov
2014-07-08 01:59:08
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!