Sprite Shootout Contest Thread

[kappa]

splitting off from this thread.

The aim of the contest is simple, to find out which is the fastest and bestest java sprite engine/library in the world

The first benchmark will be a simple test, draw as many sprites as possible while running at a constant 60 frames per second. The engine/library that is able to draw and handle the most sprites wins.

You can use the following sprite to begin with on a white background.

The balls can overlap, the transparent bits on the image must be transparent (e.g. blending is one way to achieve that), the balls must not leave the screen (should bounce of edge). When adding balls just add them at a random position, moving in a random direction and speed.

EDIT: updated sprite and rules.

EDIT2: remove collision requirement as per Riven's suggestion and updated rules.

[kevglass]

Wouldn't it make more sense to time how long it took to render 10000 frames with a fixed number of sprites. Then it doesn't matter what machine you run it on you can compare the engines?

Kev

[kappa]

Wouldn't it make more sense to time how long it took to render 10000 frames with a fixed number of sprites. Then it doesn't matter what machine you run it on you can compare the engines?

Kev

smooth movement and constant frame rate including the users preception of these should be an important part of the first benchmark. Guess we can use the above method in the next round when thing like logic speed and the ability for the engine/library to manage many sprites quickly in different situations could be tested. If the results are really close between some libraries/engines we'll have as many different benchmarks until there is a single winner.

Games published by our own members! Check 'em out!

[CommanderKeith]

Cool cool cool!

Hey what about just do the bubblemark test?  http://bubblemark.com/

Show how the game engines beat the pants off flash, silverlight, javaFX, java+swing, etc.

I see that kappa must have made one using LWJGL once, but this link is broken on the bubblemark page:

http://kappa.javaunlimited.net/bubblemark/bubblemark.html

[princec]

The gloves are off!

I'll concoct a little webstart this arvo and see how it runs.

O'course, it's going to be slightly tricky to show off the awesome animation / special effect interleaving capabilities of the SPGL sprite engine but meh

Cas

[xinaesthetic]

No reason not to do bubblemark, but would it not also make sense to at least try something that involves several different sprites, perhaps some animation, etc?

[kappa]

O'course, it's going to be slightly tricky to show off the awesome animation / special effect interleaving capabilities of the SPGL sprite engine but meh

Any photoshop/gimp/etc experts around that can help clean this 60 frame gif into a nice usable png spritesheet for the contest, hopefully with an alpha background that doesn't have horrid dots around the globe? or if you have another nice round 60 frame animatable image

Show how the game engines beat the pants off flash, silverlight, javaFX, java+swing, etc.
I see that kappa must have made one using LWJGL once, but this link is broken on the bubblemark page

Yeh it did beat/kill all the competition there, sorry bout the link, will check tonight if i still have it locally, if so will upload it again.

[Riven]

splitting off from this thread.
The balls must move and collide (bounce off) each other and the edges of the screen. The balls must not overlap unless there are too many balls to fit on the screen.

With a lot of sprites on the screen, this skews the results to the most highly optimized collision-detection algorithm... which has nothing to do with the performance of a sprite engine.

[kappa]

With a lot of sprites on the screen, this skews the results to the most highly optimized collision-detection algorithm... which has nothing to do with the performance of a sprite engine.

Ah good point, so how do you suggest we handle this? no collision at all or a standard algorithm that everyone must use? or something else? One concern is not to let the contest tests get too low level (i.e. just forcing every one to use pure java/opengl to optimise directly for the test) but to leave some bits for the actual libraries/engines running on such things to compete on how efficient they are.

[Riven]

With a lot of sprites on the screen, this skews the results to the most highly optimized collision-detection algorithm... which has nothing to do with the performance of a sprite engine.

Ah good point, so how do you suggest we handle this? no collision at all or a standard algorithm that everyone must use? or something else? my main logic behind suggesting that was not to let the contest get too low level (i.e. just forcing every one to use pure java/opengl) but to leave some bits for the actual libraries/engines running on such things to compete on how efficient they are.

If you want competitive performance, the only option is OpenGL anyway.

Besides that, I think what we really want it something that does a little more than sprite rendering, say, it has to support multiple layers. Your engine might be able to analyze a scene and determine that the 'terrain layer' is mostly static, and may be baked into a single texture. Obviously every once in a while the terrain-texture will change, as to invalidate any clever optimizations.

I think the best way to turn this into a contest is this:

• code a rolling demo, where every sprite has a deterministic position (transformation-matrix?) for each frame.
• feed all changes through an interface, that the dev will implement
  
  • public void addSprite(Sprite s, int layer)
  • public void updateSprite(Sprite s, int layer, Point p, float scale, float rotation) ?
  • public void updateSprite(Sprite s, int layer, Matrix4 m) ?
  • public void removeSprite(Sprite s, int layer)
  • public void renderFrame()
• The viewport size will be fixed (say 800x600)
• After 10.000 frames or so, the demo is over.
• Scoring: (timestampAfter-timestampBefore) (lower is better)

keeping in mind that this will have wildly varying performance characteristics due to hardware and software differences.

The idea is that everybody's output will look exactly the same.

Now somebody should write an interesting rolling-demo

Games published by our own members! Check 'em out!

[kappa]

ah nice.

just to get things rolling, lets stick with first benchmark suggestion for the first run. Mainly since its simple and doesn't require more then few minutes of coding to implement and just have an option to turn off collision detection.

[badlogicgames]

Yeah, so, even the simple benchmark rules are pretty much useless. How big should the viewport be? (fill-rate). Should blending be enabled or not? Should stuff rotate/scale? Riven's suggestion makes the most sense. Provide a simple testbed driver and let people hook it up to their stuff.

This is serious!

[Alan_W]

You can use the following sprite to begin with on a white background.

Collision detection is quite easy with circular sprites.  Why not make it really difficult with an irregular based sprite with per-pixel collision detection.  One might be able to do something with stencils.  It would certainly make it interesting.

Edit: Found some examples, but apparently the performance is poor, although I guess this means versus a rectangular bounding box.

[kappa]

anyway updated rules, no more collision requirement and balls can overlap (possibly allowing interesting optimisations).

[kappa]

Since no one has posted yet, I'll start with the first example.

You can try a JWS version here.

I'm using a nightly build of Slick2D with LWJGL 2.7.1.

It was just a quick mock up so likely not optimal and is mostly just immediate mode calls to OpenGL, so doesn't even touch the fancy stuff possible with OpenGL.

Use the 1-9 keys to change the number of balls.
Use the + and - keys to continue adding/removing balls (100 at a time) until the fps reaches about 60fps and adding any more balls causes it to go below 60, this will be the max power of the code on your computer.
Use the V key to enable/disable Vsync.

Source code is below:
ShootOut.java
Ball.java

Just for the hell of it and since CommanderKeith asked earlier about the bubblemark stuff, an Applet version can be found here. None of the other browser techs on the bubblemark page can even touch this sort of raw performance

On my machine it can draw 28,000 balls before it starts to drop below 60fps, I think the test might be too weak for modern computers

[Spasi]

Two questions from me:

- Will there be a minimum requirement for the target hardware? Personally I think the contest will be much more interesting if GLSL usage is allowed.

- Are entries that are not backed by an engine/library allowed? Meaning, a simple sprite renderer written specifically for this contest.

[appel]

I run at 22K balls in my system. I get only 12K in yours Kappa.

http://gamadu.com/games/ballsdemo/ballsdemo.jnlp
http://gamadu.com/games/ballsdemo/src/

Press spacebar to produce 1K balls.

Powered by Artemis
http://gamadu.com/artemis/

[kappa]

Two questions from me:

- Will there be a minimum requirement for the target hardware? Personally I think the contest will be much more interesting if GLSL usage is allowed.

- Are entries that are not backed by an engine/library allowed? Meaning, a simple sprite renderer written specifically for this contest.

There are no real strict rules as long as its not cheating or unfair to others, so GLSL usage is allowed.

An engine/library would have been nice but its not a requirement and I'm sure the others wouldn't mind seeing how much more they can tune their libraries/engines or even attempt to beat/match it.

After all the outcome is for geek value anyway

[Nate]

Can we change the goal? Let's make the goal 50k sprites, and the highest FPS wins. Also let's standardize the format people use to report results.

Implemented with libgdx:
http://esotericsoftware.com/spriteshootout/run.jnlp
Press space to add 1000, and up/down arrows to go by 100. See the console for FPS and ball count. This also runs on Android.

Source:
http://esotericsoftware.com/spriteshootout/spriteshootout.zip

libgdx:
http://code.google.com/p/libgdx/

I'll try on my GTX275 later, but on this crappy Nvidia 3100m I get:
libgdx: 10,800 sprites @ 60fps
appel: 10,300 @ 60fps
kappa: 10,616 @ 60fps
I guess CPU limited?

[kappa]

oh just a warning, don't run these tests for a long period of time (like going away to make a coffee while it runs), unless you want to see your gpu go up in flames

The LibGDX version is highly dangerous in that regard

[badlogicgames]

System: nvidia gtx 285, latest drivers, intel core 2 duo q9550 @2.85 ghz, java 1.6.0_22, 64-bit, client

slick(kappa): 19116 @ 60fps
slick(apple): 28300 @ 60fps
libgdx: 110000 @ 60fps

I still think this microbenchmark is flawed. We should do what Riven said, and also target a sprite count instead of an fps count. I also believe that a specialized renderer for the task at hand could outperform libgdx. But that wouldn't be a generic solution useable in a game i'd say (and if it was i'd like to integrate it in libgdx )

[badlogicgames]

edit: stupid, stupid me turns out i can't beat SpriteBatch after all...

[Nate]

My GTX275 looks the same as badlogic's 285:


Now we wait for Cas. Bring it!

[CommanderKeith]

Just for the hell of it and since CommanderKeith asked earlier about the bubblemark stuff, an Applet version can be found here. None of the other browser techs on the bubblemark page can even touch this sort of raw performance

Thanks, yep that's a feather in the cap for lwjgl and java.

One thing about that applet which is a bit weird tho is that the sprites don't appear to move smoothly. I'm testing on a computer at uni which I don't have admin access to so I can't provide the system specs sorry. I'll test at home and see what it's like.

[zammbi]

Intel i7 1.7GHz (Quad), Nvidia GT 435M, 6 gigs ram.

libgdx - 17800 balls at 60fps
Artemis - 16400 balls at 60fps
Slick2D - 12700 balls at 60fps

[appel]

On work computer:

Artemis: 6900 @ 60fps
Kappa: 6900 @ 60fps
libgdx: 6900 @ 60fps

Same performance in all!
Got NVidia Quadro FX570, Duo Core CPU.

[badlogicgames]

Everyone wins

[Riven]

Obviously these demos are fillrate limited: the code of the sprite engine has barely any impact on the framerate.

[badlogicgames]

It depends on the GPU. The mobile Nvidia crap has some very strange performance characteristics. Apple and Kappa also posted some of the results on IRC. GPUs were 8800 GT and 9600 GT, showing as big a gap as the tests by Nate and myself.

(12:26:42 AM) kappaOne: libgdx hits 45k balls before fps starts to drop
(12:26:45 AM) kappaOne: highly impressive
(12:27:13 AM) ***kappaOne closes all graphics intensive apps to allow gpu to cool down
(12:27:16 AM) badlogic: what's the number of the other benchmarks?
(12:27:27 AM) badlogic: *for
(12:27:28 AM) badlogic: fme
(12:27:32 AM) kappaOne: 28k on kappa's, 24k on appels


(12:25:05 AM) badlogic: libgdx: 8700 @60fps
(12:25:05 AM) badlogic: slick (kappa): 3016 @60fps
(12:25:05 AM) badlogic: slick (apple): 3300 @60fps
(12:25:09 AM) badlogic: nvidia ion 2
(12:25:16 AM) badlogic: atom cpu
(12:25:22 AM) badlogic: single core, 32-bit vm, client

As i said earlier, the bench is pretty much useless in any case Fun, none the less.

[Alan_W]

This laptop's only got Intel embedded graphics: libdx:4000ish, slick:3500ish
My old Mac Powerbook G4 was even slower, even though it has a proper GPU (Can't remember what offhand)

Edit: I've been thinking about how I'd write a fast sprite library.  Initial thoughts centred around vertex arrays, then moved to VBOs.  However if the demo moves every sprite a bit every frame, then I'd only use each VBO once, which seems pointless.  Maybe I could reduce the amount of data to transfer with a custom shader with a parameter list that takes x,y.  I'd still need to supply this per vertex, which would be an overhead if I used textured quads.  Maybe an indexed parameter list.  Maybe look at glDrawPixels instead, although most cards are optimised for 3D functions, so probably textured quads would be faster.

Maybe a custom shader that takes velocity/time of last position set as a parameter list and gets time from somewhere (another parameter list or write to a sub of this one.  The shader than calculates position(clip space) = position(object space) + parameter(velocity) * (parameter(current time) - parameter(vertex position timestamp).  Then I'd only need to send time each frame and update sub parts of the VBO and parameter list where a sprite had changed velocity.  In an ideal world I would send delta time and write the updates directly into the VBO in the shader, but I don't think I can do this directly; might be able to cheat round this.

Trouble is I'm not sure whether the bottleneck will be fill-rate, or CPU-GPU vertex transfer (which will be a lot with a silly number of sprites).  I've only occasionally used opengl and that in immediate mode, so don't really know what would be best performance wise.