Java-Gaming.org    
Featured games (79)
games approved by the League of Dukes
Games in Showcase (477)
Games in Android Showcase (107)
games submitted by our members
Games in WIP (535)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1] 2
  ignore  |  Print  
  JEP for making Unsafe a public API  (Read 2152 times)
0 Members and 1 Guest are viewing this topic.
Online Roquen
« Posted 2014-04-28 13:03:54 »

https://groups.google.com/forum/?hl=en-GB#!forum/jep-off-heap
Offline Drenius
« Reply #1 - Posted 2014-04-28 17:00:09 »

What?
Online Roquen
« Reply #2 - Posted 2014-04-28 17:07:45 »

JEP = Java enhancement proposal.  Unsafe = sun.misc.Unsafe a useful class that isn't public.  This is a forum for discussing making it public.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #3 - Posted 2014-04-28 17:08:43 »

This was talked about a few months ago. You could fill out a form regarding how you used Unsafe, and what additional features were required for future functionality.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline SHC
« Reply #4 - Posted 2014-04-28 17:10:38 »

Never heard of this before. Can you please explain it to me? What uses does it has? If it's private, why not use reflection to access it?

Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #5 - Posted 2014-04-28 17:15:07 »

It's not about accessibility. It's about guaranteed availability.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline pjt33
« Reply #6 - Posted 2014-04-28 22:02:49 »

What uses does it have?
Dirty hacks. I've seen it used to do custom serialisation, exploiting its ability to instantiate an object without calling its constructor and to set the values of its final fields.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #7 - Posted 2014-04-28 22:35:45 »

What uses does it have?
Dirty hacks. I've seen it used to do custom serialisation, exploiting its ability to instantiate an object without calling its constructor and to set the values of its final fields.
Or my latest hacky project: https://github.com/riven8192/LibStruct - true stack allocated structs - wouldn't have been possible without sun.misc.Unsafe

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online Roquen
« Reply #8 - Posted 2014-04-28 23:24:11 »

A heck of alot of high performance libraries use Unsafe.  You can reduce GC load by moving data off-heap.  etc. etc.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #9 - Posted 2014-04-29 08:42:18 »

You can load GC load by off-heap.  etc. etc.
Eh Huh

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Online Roquen
« Reply #10 - Posted 2014-04-29 10:01:24 »

Faulty network connection between brain and hands...corrected.
Offline BloodShura

Senior Newbie


Exp: 2 years



« Reply #11 - Posted 2014-05-05 03:36:05 »

I don't get it... I can access Unsafe normally, it's a public class. I can normally use 'Unsafe.getInstance()'. So, of what are you talking about?
Offline SHC
« Reply #12 - Posted 2014-05-05 03:37:25 »

Did you try that? It'll give you a Security exception.

Offline BloodShura

Senior Newbie


Exp: 2 years



« Reply #13 - Posted 2014-05-05 03:40:56 »

Oops, sorry, didn't noticed that!

Well, I know that you are not talking about accesibility... but in case anyone want it, you can easily access the Unsafe class...

1  
2  
3  
4  
5  
Field field = Unsafe.class.getDeclaredField("theUnsafe");

field.setAccessible(true);

return (Unsafe) field.get(null); // Return the Unsafe instance
Offline BurntPizza
« Reply #14 - Posted 2014-05-05 05:39:17 »

I don't get it... I can access Unsafe normally, it's a public class. I can normally use 'Unsafe.getInstance()'. So, of what are you talking about?

It's not that you can't access it, it that can you always guarantee that you can access it? Currently Unsafe resides in sun.misc, and is not guaranteed to be present in any given JVM as sun.misc is not a part of the standard library. This JEP proposes making Unsafe a part of the standard library.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #15 - Posted 2014-05-05 08:03:54 »

And in Android Unsafe lacks methods to read and write single primitives at abritrary addresses: putFloat, getFloat, etc. Making the API official forces every vendor to expose the same functionality.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online Roquen
« Reply #16 - Posted 2014-05-05 09:48:19 »

Wild hair thought.  Does android have the equivalent of Unsafe.defineClass ?  Thinking: Allocate a large off-heap array (which probably lies about its size for range-checking).  Then define a helper class with static methods 'like":

 public static float[] asF(Object obj) { return (float[])obj; }


which in JVM bytecodes is:
aload_0; areturn;
.  with the checkcast removed (aka now illegal and the reason a mechanism to load a class without verification).  The bodies of all 'type conversions" are the same.  You need the android IR equivalent.  You can now access the data in the array without using the prim setters/getters.

(Humm...or just one: public static <T> T asT(Object obj) {  return (T)obj; } (EDIT: This doesn't work because the checking of the type is in the caller and not the method itself.)

Well if it works for off-heap, it would work for on-heap as well (within range-check limitation).
Offline cylab

JGO Ninja


Medals: 38



« Reply #17 - Posted 2014-05-05 10:21:03 »

Making the API official forces every vendor to expose the same functionality.
Unfortunately Google is not really a Java Vendor, so I don't know if this would help the situation on Android, would it?

Mathias - I Know What [you] Did Last Summer!
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #18 - Posted 2014-05-05 11:11:36 »

Thinking: Allocate a large off-heap array
When the GC moves it, you're screwed. Either way, faking raw memory access by having huge 'fake' arrays (byte[], short[], int[], float[], long[], double[]), all pointing to the same memory address, is still no solution to mapping structs to a ByteBuffer, regardless of whether the GC will pull the rug from underneath you.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online Roquen
« Reply #19 - Posted 2014-05-05 12:31:36 »

Off-heap the GC doesn't come into play.  On-heap: I can't see a reason why it wouldn't work.  You only have references which are lying about their types and not raw memory addresses.
Online Roquen
« Reply #20 - Posted 2014-05-07 10:49:14 »

A quick hack: https://github.com/roquendm/JGO-Grabbag/tree/master/src/roquen/pr0n

On heap seems to work fine (today on my version of hotspot).
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #21 - Posted 2014-05-07 11:44:06 »

My other approach is... interesting too: (typing it out - probably doesn't even compile)
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
public static long getAddressOfReference(Object ref) {
   Object[] holder = new Object[1];
   holder[0] = ref;

   long addr;
   // read address from holder[0]
  if(is32bitJVM)
      addr = unsafe.getInt(holder, OBJECT_ARRAY_DATA_OFFSET + 0) & 0xFFFF_FFFFL; // signed
  else
      addr = unsafe.getLong(holder, OBJECT_ARRAY_DATA_OFFSET + 0);

   return addr;
}

public static Object makeReferenceToAddress(long addr) {
   Object[] holder = new Object[1];

   // write address into holder[0]
  if(is32bitJVM)
      unsafe.putInt(holder, OBJECT_ARRAY_DATA_OFFSET + 0, (int)addr);
   else
      unsafe.putLong(holder, OBJECT_ARRAY_DATA_OFFSET + 0, addr);

   return holder[0];
}

public static void copyFloatArrayHeadersToAddress(float[] src, long dstAddr) {
   for(int i=0; i<FLOAT_ARRAY_DATA_OFFSET; i+=4) {
      unsafe.putInt(dstAddr + i, unsafe.getInt(src, i));
   }
}


Now make a float[] reference in off-heap memory, shared with a FloatBuffer:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
final int sizeofFloat = 4;
int sizeofMalloc = FLOAT_ARRAY_DATA_OFFSET + elements * sizeofFloat;
ByteBuffer bb = ByteBuffer.allocateDirect(sizeofMalloc);
long bbAddr = getBufferAddress(bb);

float[] dummy = new float[0];
copyFloatArrayHeadersToAddress(dummy, bbAddr);

float[] floatArray = (float[]) makeReferenceToAddress(bbAddr);

bb.position(FLOAT_ARRAY_DATA_OFFSET);
FloatBuffer floatBuffer = bb.slice().order(ByteOrder.nativeOrder()).asFloatBuffer();


1  
2  
floatArray[1337] = 13.37f;
floatBuffer.get(1337) == 13.37f;

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online Roquen
« Reply #22 - Posted 2014-05-07 13:07:18 »

My code is without question "nuts".  I was just curious if it would work at all.  Remember I was just thinking about a potential work around for android.
Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #23 - Posted 2014-05-07 13:12:48 »

Why so apologetic / defensive?

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Online Roquen
« Reply #24 - Posted 2014-05-07 14:56:45 »

I didn't mean it to sound that way.  Without any extra information it doesn't seem useful for Java/HotSpot.
Online Roquen
« Reply #25 - Posted 2014-05-08 15:23:20 »

Small usability problem with getting a raw address.  The GC must be exactly tracking references so if the instance is moved after getting the address the reconstructed reference be foo-bared.  (I just checked)  Easy enough to shore up.

(EDIT: grammar mistake)
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #26 - Posted 2014-05-08 15:50:36 »

Had some trouble with multithreaded access in this regard. In between getting the address of a heap based object and writing the data to it - literally a couple of instructions later - the objects were being moved when I added another thread in to the mix.

Cas Smiley

Online Riven
« League of Dukes »

JGO Overlord


Medals: 744
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #27 - Posted 2014-05-08 15:56:14 »

For heap-based objects, you use
Unsafe.[put|get]...(Object ref, int offset, ... value)


What did you try to accomplish? Why would you want to write into an instance with Unsafe?

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline princec

JGO Kernel


Medals: 343
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #28 - Posted 2014-05-08 16:16:48 »

Experiments...

I was trying to get some cache-friendly way of storing processed sprite vertex data actually inside the sprites themselves (as I was having to touch the sprites anyways whilst iterating through them). So I had a big chunk of vertex00, vertex01, vertex02, vertex03, etc floats in each sprite, and I'd stick a direct FloatBuffer window over them pointing at vertex00, and write the transformed data out there; next time I had to write the sprite out to a VBO I'd point the window at it and do a bulk FloatBuffer copy straight into the VBO.

Alas, when I started to add more threads, it would mysteriously glitch now and again, and after ruling out concurrency issues with the cheeky Unsafe hacks, it turned out that the underlying sprite objects were indeed moving at arbitrary moments after I'd pointed the FloatBuffer window at them. So I gave up on that idea.

Cas Smiley

Online Roquen
« Reply #29 - Posted 2014-05-09 10:27:21 »

So has anyone looked at how hotspot expands the various get/put methods after compiling?

BTW: Since I never stated my reservation about the llegal method + (edit: the tech doesn't matter) lie about type of the array is that the compiler will end up seeing after it inlines:

pointer integerArray = byteArray

so it's almost insured to eliminate 'integerArray' since it's simply an alias of 'byteArray'.  Although it has all the information to still do the right thing, we're falling into an untested situation.  However if in some instances accessing memory via an array is actually faster (like potentially where range-checks can be eliminated) then get/put, then there's a fall-back position of making the conversion more involved such that the compiler can't determine that the two pointers are aliases.  Obviously it'd be nice if it does work with the alias elimination 'cause that'd be faster.

(EDIT: Oh and I added alternate lie-about-types methods into that playground code using riven's construction and a loop to nod about the GC maybe moving it)
Pages: [1] 2
  ignore  |  Print  
 
 

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

Riven (5 views)
2014-07-29 18:09:19

Riven (4 views)
2014-07-29 18:08:52

Dwinin (7 views)
2014-07-29 10:59:34

E.R. Fleming (23 views)
2014-07-29 03:07:13

E.R. Fleming (9 views)
2014-07-29 03:06:25

pw (39 views)
2014-07-24 01:59:36

Riven (39 views)
2014-07-23 21:16:32

Riven (26 views)
2014-07-23 21:07:15

Riven (28 views)
2014-07-23 20:56:16

ctomni231 (59 views)
2014-07-18 06:55:21
HotSpot Options
by dleskov
2014-07-08 03:59:08

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:58:24

Java and Game Development Tutorials
by SwordsMiner
2014-06-14 00:47:22

How do I start Java Game Development?
by ra4king
2014-05-17 11:13:37

HotSpot Options
by Roquen
2014-05-15 09:59:54

HotSpot Options
by Roquen
2014-05-06 15:03:10

Escape Analysis
by Roquen
2014-04-29 22:16:43

Experimental Toys
by Roquen
2014-04-28 13:24:22
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!