Java-Gaming.org    
Featured games (91)
games approved by the League of Dukes
Games in Showcase (580)
games submitted by our members
Games in WIP (500)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: [1] 2
  ignore  |  Print  
  Judging panel results!  (Read 12958 times)
0 Members and 1 Guest are viewing this topic.
Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Posted 2012-03-20 22:54:46 »

Available on the little site we all love  Grin

http://java4k.com/

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Reply #1 - Posted 2012-03-20 23:08:35 »

I just want to add that I got pleasantly surprised by the quality and immensity (yes that's a word) of the games this year, I know I say that every year Wink but this time around there were many competing games for the top. I enjoyed playing them and a few really deserve a full implementation, especially the favorite of the judges Smiley Once again 4k proves itself a good game prototyping contest.

Community and judging panel almost agree on the top 10 games, maybe not the absolute order but it's clear that community voting proves itself as a pretty accurate tool.

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline ra4king

JGO Kernel


Medals: 322
Projects: 2
Exp: 4 years


I'm the King!


« Reply #2 - Posted 2012-03-20 23:18:22 »

Wow quite different results from the community voting!

Quite funny how demonpants gave the highest scores while pjt33 gave the lowest Grin

Also I was sad to see Bubo4K all the way down there due to Drabiter not rating his own game! That 0% made his score plummet Cry

Oh and you forgot to add 2012 to the "results" page Smiley

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline kappa
« League of Dukes »

JGO Kernel


Medals: 70
Projects: 15


★★★★★


« Reply #3 - Posted 2012-03-20 23:21:14 »

nice, a lot closer to what I'd have picked compared to the community results.
Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Reply #4 - Posted 2012-03-20 23:33:20 »

Oh yes, big applause to the judges, pjt33, drabiter and Demonpants. It's a tough job going through so many games and giving them a fair shake.  Smiley

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline jimeowan

Junior Member


Medals: 1
Projects: 1



« Reply #5 - Posted 2012-03-20 23:56:23 »

Wow, 1st place, really? I didn't expect this at all, I'm glad you liked my game, thanks!

Indeed I felt like The Little Scientist might be a good start for a real game... If I find the time maybe I'll start making a mobile app out of it, who knows Smiley

Anyway thank you Appel for setting up this contest, I learned a lot of things by taking part to it. See you next year Wink

Author of "The Little Scientist" (judging panel winner for Java4K 2012 </brag>)
Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Reply #6 - Posted 2012-03-21 00:08:21 »

Wow, 1st place, really? I didn't expect this at all, I'm glad you liked my game, thanks!

Indeed I felt like The Little Scientist might be a good start for a real game... If I find the time maybe I'll start making a mobile app out of it, who knows Smiley

Anyway thank you Appel for setting up this contest, I learned a lot of things by taking part to it. See you next year Wink

Congratulations Smiley

If you go through all the winning games in the past java4k contests, then you'll realize all of them have one thing in common, the potential of being a full game. 4k is great for prototyping games, and as a judge what I love the most is seeing potential, my imagination takes off and I have all sorts of ideas for the game.

That said I'm pleased you were #1, and I hope you try to think about what you can do next Smiley

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline Morre

JGO Knight


Medals: 2
Projects: 10


I'm Dragonene on IRC.


« Reply #7 - Posted 2012-03-21 00:57:53 »

I for one feel it was a well deserved victory, jimeowan Smiley

Great work, judges! Lots of games to go through - you did well!

I'm also surprised and pleased by the reviews on my entry.

Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Reply #8 - Posted 2012-03-21 01:03:11 »

Next contest will be the 10th, X. A whole whopping decade of 4k delight. Maybe we should do something special for the grand anniversary contest.

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline dapy
« Reply #9 - Posted 2012-03-21 01:39:21 »

A big THANK YOU to the judges, and especially to Appel, for their hard work.

Congratulations to jimeowan - The Little Scientist was an excellent and deserving winner!
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Damocles
« Reply #10 - Posted 2012-03-21 05:59:10 »

Thanks for all the time the judges invested to review ALL submissions.

Also gives me good feedback for my games on what to improve.

Offline StephR

Senior Member


Medals: 9



« Reply #11 - Posted 2012-03-21 07:39:30 »

Congratulations, jimeowan ! The Little Scientist had all the qualities to get this first place. I was very impressed by the investment from our judges to review all our games - and they even didn't get paid for that ! I was also impressed by the number of top technical quality, innovative and especially non violent games, all made in 4 Kb. My own not included ! I don't know if it really deserved it, but I was surprised and glad to discover how it was rated.

And of course some special thanks to appel for organizing flawlessly the contest once more. Count on my participation for the 10th edition.
Offline Archibald

Junior Member


Projects: 1



« Reply #12 - Posted 2012-03-21 11:44:43 »

Awesome... I got community rating, judges rating, 4 different reviews... It is dreamland of game's feedback Smiley Thanks.

But as they say, offer a finger and they will take a whole arm, so, I would like to propose an addition of community vote points breakthrough (how many people voted 5, how many voted 1). This would be benefitial in order to determine if your game was mediocre (a lot of people giving 1-3) or niche (a few people but giving 4-5). It could be added as a link when clicking the currently existing score number.

Offline ReBirth
« Reply #13 - Posted 2012-03-21 12:06:20 »

I give big applause to jimeowan for his little scientist!

I've been following java4k for 2 years. In past I neither submitting or what, just came to play around with the games. Yes I agree with appel that this year is awesome, thanks to you guys!  Cool

In term of score, mine and demonpants's are really (almost) identic. Pjt33's reviews are really interesting. They're long and very detailed. And funny. I see how he feared when my entry was going to break his speaker from the review. Among reviews, those mine are the shortest as you see. I might lazy sometimes but my biggest concern is grammar. English is my third language and I don't want my reviews looks too bad.

Also I was sad to see Bubo4K all the way down there due to Drabiter not rating his own game! That 0% made his score plummet Cry
Thanks for your attention Wink I know the risk when I applied for judge position, although it's hard to since it's my first entry. I did give community voting to myself, and it was 2 points in selfishly way.

FYI, that "drabiter" guy is me. "Rebirth" as username is too mainstream I think.

Offline StephR

Senior Member


Medals: 9



« Reply #14 - Posted 2012-03-21 13:40:21 »

It seems there is a small bug on the 2012 JUDGING PANEL WINNERS at http://www.java4k.com/. Some games are not well sorted on their score :

Place #1 with 96.3% score
Place #2 with 93.3% score
Place #3 with 91.7% score
Place #4 with 90.8% score
Place #5 with 91.4% score
Place #6 with 91.7% score
Place #7 with 89.2% score
Place #8 with 86.6% score
Place #9 with 85.3% score
Place #10 with 86.8% score
Offline Riven
« League of Dukes »

JGO Overlord


Medals: 605
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #15 - Posted 2012-03-21 13:52:09 »

I demand a recount. Kiss



In Soviet Russia, game rates you! Emo

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline ReBirth
« Reply #16 - Posted 2012-03-21 14:03:00 »

Only in front page, on table they're right. Hope appel fix it asap.

Offline Riven
« League of Dukes »

JGO Overlord


Medals: 605
Projects: 4
Exp: 16 years


Hand over your head.


« Reply #17 - Posted 2012-03-21 14:07:20 »

Only in front page, on table they're right. Hope appel fix it asap.
The problem is that if there are two different listings, you don't know which is right.

Hi, appreciate more people! Σ ♥ = ¾
Learn how to award medals... and work your way up the social rankings
Offline RussS

Junior Member


Projects: 3



« Reply #18 - Posted 2012-03-21 20:51:39 »

Thanks to all the judges! This was great fun and I can't wait till next year Smiley
Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Reply #19 - Posted 2012-03-21 21:59:02 »

Yea, sorry about the mis-ordering on the front page. The page with the entire score table is the correct one. Strange though, the front page listing uses the same service to retrieve the scores as the score table page. :/ Will fix when I figure out what's wrong... bad php!

FIXED!

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Online pjt33
« Reply #20 - Posted 2012-03-22 00:30:47 »

Wow quite different results from the community voting!
It was a lot closer than I recall in previous years, which have tended to have a clear winner. This year every single judge ranked a different game top, and the community voting yet another one. So congratulations especially to Marwane Kalam-Alami for The Little Scientist and zeroone for Laser Pinball, but also to 3darray for Nameless, Russell A Spitzer for 4Khords, dapy for Abstract Glowy Vector Wars, and Damocles for Cave4k.

Oh, Marwane, nice reference to Saint-Exupéry. I picked up on it after writing my review, and figured that someone else would be sure to mention it, but since no-one has I thought I'd let you know that it didn't go entirely unnoticed.

Quite funny how demonpants gave the highest scores while pjt33 gave the lowest Grin
You know how magazines tend to rate everything 7/10 or higher? I was deliberately trying to avoid that and use something nearer the full range of points available, and given how long a sequence there is of games which are separated only by 1% in my ranking I'm glad I did. Appel has previously suggested that rather than giving a percentage score the judges should just bin the games roughly and then order them within the bins, and now that I've seen it from the judging side I think that suggestion has a lot of merit.

Also I was sad to see Bubo4K all the way down there due to Drabiter not rating his own game! That 0% made his score plummet Cry
Not to mention that it mucks up my spreadsheet to calculate correlations between everyone's rankings. Wink

Seriously, though, it does seem a bit low. Appel, didn't you make the scoring system ignore 0s after the brouhaha a couple of years ago?

In term of score, mine and demonpants's are really (almost) identic.
Your rankings have a Pearson rank correlation coefficient of 0.60, which is the highest of any pairing of judges but not fantastically high. In terms of individual judges vs community, Appel is at 0.69, followed by you at 0.67. Overall, though, the combined judges' ranking vs the community ranking has a PRCC of 0.73.

Pjt33's reviews are really interesting. They're long and very detailed. And funny.
Thank you. Cheesy

I now know why teachers' reports are always full of clichés. It's a real effort to avoid repeating yourself when writing 50 reports.

One final comment: I had Linux compatibility problems with more than 10% of the games. This may in part be Oracle's fault, but note that most games didn't have problems. I highly recommend reading the discussion threads from previous years on applet templates.
Offline appel

JGO Wizard


Medals: 49
Projects: 5


I always win!


« Reply #21 - Posted 2012-03-22 01:22:19 »

Drabiters Bubo4K game shouldn't have been reviewed by the judges, thus it shouldn't have received any score.

Quote
Due to the fact there is community voting, judges are free to submit a game into the contest.

    Note: These games will only be voted on in the community vote. They will not be considered by the judging panel, and therefore will receive no score from the judging panel. Community voting is separate and independent from the judging panel, and cannot affect or influence each other. This allows judges to submit games for community voting without worrying about any conflict of interest.
Rules: http://java4k.com/index.php?action=view&page=rulesjudg

My system doesn't yet allow making exceptions like these, haven't added it because there's never been a need for it. So, his game slipped through, I didn't even think of it when reviewing the games Smiley But no biggie, I see no reason to remove the game from the list, especially when it's so low. Shocked



Quote
Appel has previously suggested that rather than giving a percentage score the judges should just bin the games roughly and then order them within the bins, and now that I've seen it from the judging side I think that suggestion has a lot of merit.
I had forgotten about that Smiley Percentages seem to work alright, but perhaps too "absolute". I mean what's the difference between 58% and 63%? I do prefer bucketing the games into buckets like:
1) Superb!
2) Excellent
3) Good
4) Decent
5) Mediocre
6) SO HORRIBBLE MY EYEESS BLEEEEEDD!! (ok maybe not)

Once that process is done, judges can prioritize games within each bucket. Bucketing the games like this is super fast.

Problem is... assigning the score isn't the most time consuming task, it's actually playing the game and then write a review. Also, what if one judge decided to put one game into Superb! bucket but another into Mediocre? I don't like a whole lot of obscurity in how the score is calculated, e.g. everyone understands percentages and normalized averages. So, my idea to solve that would for the judges to decide on which games go into which bucket together, but then individually order them within the bucket. Debatable. But this is similar to what I do at work in SCRUM, we all as a group try to reach an agreement of how much work a story is. But that would introduce another issue though... some don't want judges to coordinate the results. I admit, I like the "surprise" factor, "oh, that's how the others voted and wow, that's the winning game?"-feeling.

However, there's another practical problem. I'd say around 50 games is the upper limit of what a judge can be expected to review, if we get 100 games I wouldn't know what to do, it'd take a full week to review probably.. 20 games a day, and with each game taking maybe 5-10 minutes it's around 2-3 hours per day. With buckets the judges could only write a review for the games in top 3 buckets and skip the lower buckets.

Buckets solve a few problems, but don't necessarily satisfy everything or everyone.

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline ra4king

JGO Kernel


Medals: 322
Projects: 2
Exp: 4 years


I'm the King!


« Reply #22 - Posted 2012-03-22 03:56:24 »

Oh, Marwane, nice reference to Saint-Exupéry. I picked up on it after writing my review, and figured that someone else would be sure to mention it, but since no-one has I thought I'd let you know that it didn't go entirely unnoticed.
What Saint-Exupéry reference? Clueless

In term of score, mine and demonpants's are really (almost) identic.
Your rankings have a Pearson rank correlation coefficient of 0.60, which is the highest of any pairing of judges but not fantastically high. In terms of individual judges vs community, Appel is at 0.69, followed by you at 0.67. Overall, though, the combined judges' ranking vs the community ranking has a PRCC of 0.73.
I love how this makes absolutely no sense to me Grin

I now know why teachers' reports are always full of clichés. It's a real effort to avoid repeating yourself when writing 50 reports.
I know the feeling exactly Smiley

Offline bysse

Senior Member


Medals: 3
Projects: 1



« Reply #23 - Posted 2012-03-22 04:28:11 »

Thanks for the reviews and a big hand to Marwane, awesome game!

I realize now that i focused too much on the music,
forgetting to test the game properly which in turn lead to buggy physics on some platforms, damn Smiley

Offline StephR

Senior Member


Medals: 9



« Reply #24 - Posted 2012-03-22 08:37:39 »

What Saint-Exupéry reference? Clueless

The novel The Little Prince tells the story of a boy who travels from planet to planet, meeting different silly adults ( although no Java programmer if I remember it well ). He finally arrives to Earth where he meets the story's narrator.

I hadn't noticed the link, but now I find it quite obvious.
Offline StephR

Senior Member


Medals: 9



« Reply #25 - Posted 2012-03-22 08:47:32 »

I should also add that The Little Scientist's graphics are reminiscent of the novel's original illustrations, which were also made by its writer,
Antoine de Saint-Exupéry.
Offline jimeowan

Junior Member


Medals: 1
Projects: 1



« Reply #26 - Posted 2012-03-22 10:27:31 »

Oh, Marwane, nice reference to Saint-Exupéry. I picked up on it after writing my review, and figured that someone else would be sure to mention it, but since no-one has I thought I'd let you know that it didn't go entirely unnoticed.

Haha congrats, you're the first one who mentioned the reference  Grin

I realize now that i focused too much on the music,
forgetting to test the game properly which in turn lead to buggy physics on some platforms, damn Smiley

Then I guess I'm lucky, I didn't bother at all testing my applet in various environments... But luckily I had to work on the game alternatively on Linux & Windows x)



Now regarding the ranking system, even if I understand that the current way to do it is time consuming for reviewers, I totally agree with Appel in that using 'bins' would make the ranking a bit shady for others. Now, I guess that what takes a lot of time is to write the reviews, so what about splitting the games between the judges? They would still have to give grades to everyone (else it wouldn't be fair), but only write reviews for a subset. For instance, 2 reviews per game would be fair enough.

Another (smaller) suggestion: an alternative to the idea of bins would be to give notes between 0 and 10, they can easily be converted to percents once averaged, and are probably faster to choose.

Author of "The Little Scientist" (judging panel winner for Java4K 2012 </brag>)
Online pjt33
« Reply #27 - Posted 2012-03-22 10:45:00 »

Quote
Appel has previously suggested that rather than giving a percentage score the judges should just bin the games roughly and then order them within the bins, and now that I've seen it from the judging side I think that suggestion has a lot of merit.
I had forgotten about that Smiley Percentages seem to work alright, but perhaps too "absolute". I mean what's the difference between 58% and 63%? I do prefer bucketing the games into buckets like:
1) Superb!
2) Excellent
3) Good
4) Decent
5) Mediocre
6) SO HORRIBBLE MY EYEESS BLEEEEEDD!! (ok maybe not)

Once that process is done, judges can prioritize games within each bucket. Bucketing the games like this is super fast.

Problem is... assigning the score isn't the most time consuming task, it's actually playing the game and then write a review. Also, what if one judge decided to put one game into Superb! bucket but another into Mediocre? I don't like a whole lot of obscurity in how the score is calculated, e.g. everyone understands percentages and normalized averages. So, my idea to solve that would for the judges to decide on which games go into which bucket together, but then individually order them within the bucket.
The scoring system could work on the basis of ranks: the game I consider the best gets n points, the next gets n-1, and the worst gets 1. Then average over the judges. It has normalisation built in automatically, and from a practical point of view it's not so far off how I ended up scoring with all the 1% differences between games. The finesse you lose is the ability to have larger gaps between games at the two extremes.

Collaborating would make things take much longer: the judges would have to be all online at the same time, or it would spread over several days. In cases where one disagrees it might be easy to reach a quick compromise, but there were some games this year which split the judges 2-2 (Dawn, Drift Driver, Electron Golf, Fleazy, and Galactic Conquest being the clearest). Given the time pressure they're already under, the stubbornest judge would have disproportionate influence.

Now, I guess that what takes a lot of time is to write the reviews, so what about splitting the games between the judges? They would still have to give grades to everyone (else it wouldn't be fair), but only write reviews for a subset. For instance, 2 reviews per game would be fair enough.
That's not a bad idea.

Another (smaller) suggestion: an alternative to the idea of bins would be to give notes between 0 and 10, they can easily be converted to percents once averaged, and are probably faster to choose.
When you're judging 50 games and want to express an opinion on their relative merit you want at least 20 different scores to give them. Ranking 0-10 would mean giving 0 to some games just to get the differentiation, which would be quite discouraging for the people who made those games, and a lot of post-review agonising over games on the borderline between 6 and 7.
Offline Damocles
« Reply #28 - Posted 2012-03-22 12:38:40 »

Quote
Quote from: jimeowan on 38 minutes ago
Now, I guess that what takes a lot of time is to write the reviews, so what about splitting the games between the judges? They would still have to give grades to everyone (else it wouldn't be fair), but only write reviews for a subset. For instance, 2 reviews per game would be fair enough.
That's not a bad idea.

Given that judges have a personal difference in scoring, this would distort the results.
Every judge has his own personal "metric".

If 2 judges are more critical (give lower scores on average) and the other 2 give higher scores,
the the first batch of games would have a lower expected ranking than the second by expectation.
Thus comparing the scores would be like comparing oranges with apples.

Its just a different testbase then.
I dont see a way to realistically compare scores without each judge to judge all games.

Offline jimeowan

Junior Member


Medals: 1
Projects: 1



« Reply #29 - Posted 2012-03-22 13:46:38 »

When you're judging 50 games and want to express an opinion on their relative merit you want at least 20 different scores to give them. Ranking 0-10 would mean giving 0 to some games just to get the differentiation, which would be quite discouraging for the people who made those games, and a lot of post-review agonising over games on the borderline between 6 and 7.

I guess you're right, if no one dares to give scores below 5 it's no finer than the bin system...

Given that judges have a personal difference in scoring, this would distort the results.
Every judge has his own personal "metric".

I was only suggesting to split the written reviews, not the scoring. Else the rankings would not make sense indeed.

Author of "The Little Scientist" (judging panel winner for Java4K 2012 </brag>)
Pages: [1] 2
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

xsi3rr4x (48 views)
2014-04-15 18:08:23

BurntPizza (44 views)
2014-04-15 03:46:01

UprightPath (60 views)
2014-04-14 17:39:50

UprightPath (42 views)
2014-04-14 17:35:47

Porlus (58 views)
2014-04-14 15:48:38

tom_mai78101 (81 views)
2014-04-10 04:04:31

BurntPizza (140 views)
2014-04-08 23:06:04

tom_mai78101 (240 views)
2014-04-05 13:34:39

trollwarrior1 (200 views)
2014-04-04 12:06:45

CJLetsGame (207 views)
2014-04-01 02:16:10
List of Learning Resources
by SHC
2014-04-18 03:17:39

List of Learning Resources
by Longarmx
2014-04-08 03:14:44

Good Examples
by matheus23
2014-04-05 13:51:37

Good Examples
by Grunnt
2014-04-03 15:48:46

Good Examples
by Grunnt
2014-04-03 15:48:37

Good Examples
by matheus23
2014-04-01 18:40:51

Good Examples
by matheus23
2014-04-01 18:40:34

Anonymous/Local/Inner class gotchas
by Roquen
2014-03-11 15:22:30
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!