Java-Gaming.org Hi !
Featured games (90)
games approved by the League of Dukes
Games in Showcase (726)
Games in Android Showcase (216)
games submitted by our members
Games in WIP (796)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1]
  ignore  |  Print  
  Java Native I/O v 1.0: beta release  (Read 3857 times)
0 Members and 1 Guest are viewing this topic.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Posted 2017-02-05 12:53:59 »

I started a project with the goal to provide access to native data (ByteBuffer) through an interface provided by the application.

You probably know the hassle to write wrappers around byte buffers to access uniform blocks or data in vertex array buffers and such. Writing the same kind of code over and over makes me sick and since I have a lot of experience in code generation, I decided to put an end to it.

The idea is roughly this: You just write a class skeleton (similar to RMI for example) which you provide to a class generator at runtime. The class generator adds required functionality and information in a derived class and gives you an instance of that class to access. This instance then acts as the interface to the native data.

This will all happen at runtime, so there is no visible preprocessing step involved in your build process. It doesn't have to be at runtime, but it is possible and so most people will use it that way.

This sounds just like a tiny advantage, I know, but the bigger picture, is this: Meta information about types in data is needed at different places. Besides just accessing the data in a byte buffer, you can use the same meta-information provided by the class to declare the format of meshes, and then use the same information again to associated appropriate vertex attributes or even define them already in the class as annotations. Same for uniforms: You can identify the appropriate uniform variable in the shader using the name of the member variable or an annotation again .. and so on. It makes the whole thing more consistent.

If anybody is interested in the progress or first prototypes, here is the address: http://homac.cakelab.org/projects/org.cakelab.nio/

EDIT: I've reviewed existing approaches related to this topic: http://homac.cakelab.org/projects/org.cakelab.nio/existing.html

EDIT(22.03.17): Major change of approach after detailed analysis: http://homac.cakelab.org/projects/org.cakelab.nio/current.html

EDIT(30.04.17): First release (beta): http://homac.cakelab.org/projects/org.cakelab.nio/

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline theagentd

« JGO Bitwise Duke »


Medals: 970
Projects: 4
Exp: 8 years



« Reply #1 - Posted 2017-02-05 13:17:22 »

Do you have an example of the workflow? I'm interested.

Myomyomyo.
Offline jonjava
« Reply #2 - Posted 2017-02-05 13:17:46 »

I don't understand. What are you trying to accomplish? And why isn't JNI an option?

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #3 - Posted 2017-02-05 13:25:50 »

Yes, there is an example in the prototype, but for the runtime code generation and data access only. The integration with OpenGL is a future project.

Example would go like this

Your class:
abstract class MyData extends BufferWrapper {
    float x,y;

    public abstract float getX();
    public abstract void setX(float x);
    // ...
}


Use:

ByteBuffer nativeData = ...;
MyData data = BufferWrapperFactory.load(MyData.class, nativeData);

float a = data.getX();
data.setY(1f);



etc. ...

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #4 - Posted 2017-02-05 13:31:43 »

I don't understand. What are you trying to accomplish? And why isn't JNI an option?

JNI is no option in this case. Do you know for example LWJGL. There is a lot of data exchange with data of unkown type to be send and received between graphics card memory and Java heap space. This is all done via ByteBuffer interfaces.

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline cylab

JGO Kernel


Medals: 165



« Reply #5 - Posted 2017-02-05 13:34:27 »

Did you see http://www.java-gaming.org/topics/once-again-fast-mappedobjects-implementation/18852/view.html?

Mathias - I Know What [you] Did Last Summer!
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #6 - Posted 2017-02-05 13:43:39 »

No I haven't. Thank you for the link.

This looks pretty close, maybe I can learn a thing or two. Downside of his approach - what I've seen so far - he always copies the whole chunk. Not sure if that's what I want. 


Haven't done serious research on existing stuff yet. This prototype just took me a few hours to get running and I didn't bother researching in advance. 

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1277
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #7 - Posted 2017-02-05 14:09:11 »

LibStruct is zero-copy. You just map a struct to an existing memory pointer / buffer

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline theagentd

« JGO Bitwise Duke »


Medals: 970
Projects: 4
Exp: 8 years



« Reply #8 - Posted 2017-02-05 14:27:35 »

How does your code work? Does it actually generate a new implementation of the class when you first load() it? What kind of overhead are we talking about? Garbage? Can I control the ordering of elements? I can see this being useful for uniform buffers for example, while being a bit less intrusive compared to MappedObjects, which require a Java agent.

More questions:
 - It's very rare to read back a data structure from OpenGL at least. For writing it should only need a setX() function, as it can figure out the type of it from the argument to it. A write-only structure should then only need a setX() function for each parameter, minimizing the boilerplate code you have to write for the code generation to work. Similarly, you could have a read-only data structure which only has a getX() function, with the types gathered from the return types of those functions. Also, since MyData won't need to contain
float x, y;
with these changes, there won't be any wasted memory.
 - It'd be useful to have a way to precompile said classes so that no reflection is needed to use this.

Also, you can place your code in a code block on this forum, see the hashtag button when writing your posts.

Myomyomyo.
Offline princec

« JGO Spiffy Duke »


Medals: 904
Projects: 3
Exp: 16 years


Eh? Who? What? ... Me?


« Reply #9 - Posted 2017-02-05 14:30:48 »

I'd give up on it and live with the palette of colours we have to paint the picture within the restrictions... it's likely that this will come to the language circa Java 10/11 eventually anyway along with value types.

Cas Smiley

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #10 - Posted 2017-02-05 14:49:57 »

How does your code work? Does it actually generate a new implementation of the class when you first load() it? What kind of overhead are we talking about? Garbage? Can I control the ordering of elements? I can see this being useful for uniform buffers for example, while being a bit less intrusive compared to MappedObjects, which require a Java agent.

We are talking about plans and a prototype. Currently, class generation overhead is close to null if you consider that this class is generated just once for the life time of the application you can spend even minutes to compile stuff for the first time.

An offline generation of classes is of course possible, but that was not my goal.

I agree, having an agent (referring to LibStruct) is not my plan. But it looks very tempting to switch to that instead Cheesy

And talking about intrusiveness .. having another build step is intrusive as well .. don't you think?

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline theagentd

« JGO Bitwise Duke »


Medals: 970
Projects: 4
Exp: 8 years



« Reply #11 - Posted 2017-02-05 15:14:47 »

And talking about intrusiveness .. having another build step is intrusive as well .. don't you think?
Not if I can use the runtime compilation of the helper classes for debugging, then seamlessly switch to the prebuilt one for actual releases.

Myomyomyo.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #12 - Posted 2017-02-05 15:29:05 »

And talking about intrusiveness .. having another build step is intrusive as well .. don't you think?
Not if I can use the runtime compilation of the helper classes for debugging, then seamlessly switch to the prebuilt one for actual releases.

It will be an option of course, but the actual challenge is to get it done without it.

Do you know Java remote method invocation (RMI)? That was originally with a code generation pass (rmigen when I remember correctly) you had include in your build environment before you could use it and then they switched to runtime because people just find it too cumbersome to write a makefile. And I can understand it. For example, Eclipse is really sucky when it comes to integrating additional build steps in their build system and support of a code generation step with auto detection of files that do not exist when the build process starts and stuff in a makefile is .. erm .. lets say difficult Cheesy.

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #13 - Posted 2017-02-11 11:27:58 »

I've reviewed existing approaches related to this topic.

http://homac.cakelab.org/projects/org.cakelab.nio/existing.html


I think I've found everything about it, but if someone knows about a tool or framework I haven't mentioned, please tell me.

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1277
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #14 - Posted 2017-02-11 11:37:13 »

I've reviewed existing approaches related to this topic.

http://homac.cakelab.org/projects/org.cakelab.nio/existing.html


I think I've found everything about it, but if someone knows about a tool or framework I haven't mentioned, please tell me.

Quite an impressive review!


Quote
@StructField: Used to declare a field to be part of the struct. Fields without it will be considered to be transient in respect to the structs layout.
@CopyStruct: Used to declare methods to return a copy of the struct.
@TakeStruct: Used to declare methods which take a parameter, which is a mapped struct object.

@CopyStruct and @TakeStruct both refer to the return-value (unrelated to parameters). It is basically an instruction to the callsite, on whether to copy the struct that was returned to the head of its stack (in case it was stack allocated in the called method), or take it as is (in case the struct was malloc-ed or pre-existing, surviving the scope of the callee).

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #15 - Posted 2017-02-11 11:42:33 »

@CopyStruct and @TakeStruct both refer to the return-value (unrelated to parameters). It is basically an instruction to the callsite, on whether to copy the struct that was returned to the head of its stack (in case it was stack allocated in the called method), or take it as is (in case the struct was malloc-ed or pre-existing, surviving the scope of the callee).

So, Copy is return by value and Take is return by reference?

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline Riven
Administrator

« JGO Overlord »


Medals: 1277
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #16 - Posted 2017-02-11 13:27:22 »

Yes. Pointing

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings!
Offline Spasi
« Reply #17 - Posted 2017-02-11 15:00:00 »

I've reviewed existing approaches related to this topic.

http://homac.cakelab.org/projects/org.cakelab.nio/existing.html

Very interesting, some answers and corrections:

Quote
Offsets or member variables are stored in static final members of the struct class, which will be most likely treated as constants by the JIT compiler and as such inserted in the code (text cache not data cache) (TODO: proof)

This is correct and can be easily proven using -XX:+PrintAssembly. The work done by Layout in the static init blocks has been designed to achieve exactly this.

Quote
An associated class derived from StructBuffer is used to attach a byte buffer to instances of Struct classes. Instances of Struct classes are backed by a buffer, always.

This is all wrong:

- The ByteBuffer container field in both Struct and StructBuffer is optional. In fact, when writing "idiomatic" LWJGL 3 code, it almost always goes unused and is quite often eliminated entirely via escape analysis (nothing is written to it and no code reads from it).

- A StructBuffer is a packed array of struct values, there are no instances of Struct classes. One option is to access the array data with the flyweight pattern. The other option is to retrieve Struct class instances as you iterate over the array but, again due to how one writes LWJGL 3 code, those allocations are eliminated via escape analysis. You can use a tool like JITWatch to verify this.

- There are multiple ways to allocate either a Struct or a StructBuffer: a) backed by a ByteBuffer (which itself can be allocated in multiple ways) b) using malloc/calloc/etc c) using the MemoryStack. The choice depends on the amount of code the user is willing to write and the amount of memory management they afford to worry about. Only the first option utilizes the container field, for obvious GC reasons.

Quote
It very much looks like this is derived work from the author of LibStruct (see below) but was discarded in LWJGL 3 for yet unknown reason.

3 reasons:

a) LibStruct does a great job and should interoperate just fine with LWJGL 3.
b) LWJGL 3 does bindings only, improving user code is not one of its goals. We have specific struct types that must be supported and static code generation is the simplest solution.
c) Project Panama is also going to expose struct fields using methods (unless they do something with value types, but I doubt it). When LWJGL (4?) gets ported to Panama, hopefully existing code won't require significant changes.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #18 - Posted 2017-02-12 06:52:41 »

Yes, I see. Thanks for the explanations. I will incorporate those in the review.

And thanks for the link to Project Panama. I heard about it before but somehow forgot about it.

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #19 - Posted 2017-03-22 21:47:14 »

My approach has significantly changed, since I'm no longer aiming to support dynamic code generation. Instead I'm going to provide plugins for IDEs to trigger code generation for selected types, similar in its methodology to for example generating getter and setter methods for selected members of a class.

See update here: http://homac.cakelab.org/projects/org.cakelab.nio/current.html

(probably TL;TR)

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline gouessej
« Reply #20 - Posted 2017-03-22 23:18:54 »

You don't talk about GlueGen.

Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #21 - Posted 2017-03-24 17:53:36 »

You don't talk about GlueGen.
Thank you, gouessej, I didn't know about it. Will add it under JNI approaches.

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Offline homac

Senior Devvie


Medals: 17


life is cake.


« Reply #22 - Posted 2017-04-30 16:51:37 »

Pushed a first beta release - mostly to get some feedback.

There is a basic implementation of an eclipse plugin, which is the most comfortable way to use the code generator.

The Eclipse Example should help to get started. Detailed documentation still missing.

The guy who cooks Java .Blend and helps people to launch their Life in the Woods.
Pages: [1]
  ignore  |  Print  
 
 

 
Archive (286 views)
2017-04-27 17:45:51

buddyBro (477 views)
2017-04-05 03:38:00

CopyableCougar4 (924 views)
2017-03-24 15:39:42

theagentd (937 views)
2017-03-24 15:32:08

Rule (949 views)
2017-03-19 12:43:22

Rule (916 views)
2017-03-19 12:42:17

Rule (918 views)
2017-03-19 12:36:21

theagentd (979 views)
2017-03-16 05:07:07

theagentd (891 views)
2017-03-15 22:37:06

theagentd (688 views)
2017-03-15 22:32:18
List of Learning Resources
by elect
2017-03-13 14:05:44

List of Learning Resources
by elect
2017-03-13 14:04:45

SF/X Libraries
by philfrei
2017-03-02 08:45:19

SF/X Libraries
by philfrei
2017-03-02 08:44:05

SF/X Libraries
by SkyAphid
2017-03-02 06:38:56

SF/X Libraries
by SkyAphid
2017-03-02 06:38:32

SF/X Libraries
by SkyAphid
2017-03-02 06:38:05

SF/X Libraries
by SkyAphid
2017-03-02 06:37:51
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!