Judging Results

[pjt33]

Just do it (TM).

However you should probably also put up the following variants:

- Remove all the highest scores for each entry
- Remove all the lowest scores for each entry
- Remove all both the highest and the lowest for each entry
- Remove a particular judge's results for each entry
- Make all the scores for every game but the one selected zero (ok, maybe not this one ).

It'd hardly be fair to judge some entries on 4 judge's results and some on 5 given the disparity in the judges ranges of scores and approach to judging.

This could go on and on and on...

The way to handle the disparity in ranges of scores is to normalise. Doing that and discounting 0s gives:

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64   65   66   67   68

Sorting category Overall by Normalised means ignoring zeros  1. Left 4k Dead                            94.5  2. 4bidden Fruit                           90.1  3. Bridge4k                                88.8  4. Pixeloids4k                             88.2  5. Hunt 4k Bread                           86.4  6. Keggle                                  85.1  7. Robo rampage                            84.1  8. jm00 - a boomshine clone in 4k v1.2     84.0  =  Red Baron 4K                            84.0 10. Fortress4K                              83.5 11. Splosh4k                                82.9 12. Putty Shuffle                           82.3 13. World Rally Driver 4k                   82.0 14. Tekicars4k                              81.6 15. Just Get Bigger 4K                      81.4 16. BlockBuster4k                           80.3 17. NiGHTS 4k                               79.8 18. MEG4kMAN                                79.1 19. Q*Bert4K                                78.7 20. Conquest of Planets                     78.5 21. Invaders                                78.2 22. RevolvoMan-4K                           77.8 23. 4x4k                                    77.5 24. Jetp4k                                  76.4 25. 4bsolution                              75.9 26. Space Paranoids 2D4K                    75.6 27. Critters4k                              75.5  =  Ice Fighters                            75.5 29. F-Zero 4K                               75.4 30. Kart                                    74.6 31. J-Type                                  74.5 32. Asteroid Alert!                         74.4 33. Run Over Zombies                        74.0 34. Super Marble World                      73.2 35. Sea Spin                                72.8 36. Word Twister                            72.3 37. Sea through                             70.5 38. RhythmSphere                            69.8 39. Treasure4k                              69.5 40. submarin                                68.1 41. Bio 4K                                  67.3 42. HexoDama                                66.6 43. Honey4kKeeper                           66.1 44. Doodles                                 66.0 45. Grasshopper4k                           65.6 46. Crimsonland4k version 1.001             65.5 47. Alienz 4K                               64.5  =  Star 4x                                 64.5 49. Crack Tower Defense                     63.2 50. Gravitational Fourks                    61.2 51. Ultimate Tic-Tac-Toe                    60.9 52. Anyone 4 Tennis?                        59.8  =  PacPitfall4k                            59.8 54. Coffee Shop Puzzle                      59.0 55. Infinite Platformer 4k                  57.0 56. Frequent Flier                          56.4 57. Deathchase                              55.5 58. BlisterBall                             53.0 59. Scr4mble                                52.5 60. Dragon 4k Boxing                        52.2 61. Arena4k                                 51.9 62. Beez Helper                             51.6 63. i4kopter                                50.6 64. QuadSquad4k                             50.1 65. Virtual On 4k                           44.3 66. 4K Maze                                 37.5 67. Desert Bus                              24.7

Code to follow, because it doesn't fit in one post.

[size=5pt]One could also argue that it's a coding contest and failing with an NPE is a failure to code to some API or framework given how many other games managed to run on the same setup.[/size]

The fact that you don't know which API or framework is a hint that it's not necessarily the coder's fault. My game broke with a version of Java which was released after I finished writing it. I don't know which API or framework broke, nor in what way, because Webstart is rather sparse with its error messages. The only error message it shows seems to be an internal one.

[pjt33]

Had to split it because of message length restrictions. See the TODO.

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64   65   66   67   68   69   70   71   72   73   74   75   76   77   78   79   80   81   82   83   84   85   86   87   88   89   90   91   92   93   94   95   96   97   98   99   100   101   102   103   104   105   106   107   108   109   110   111   112   113   114   115   116   117   118   119   120   121   122   123   124   125   126   127

import java.util.*; public class Analyse4k{    private static String[] areaNames={"Overall","Technical","Presentation"};    public static void main(String[] args){       // Note: values extracted automatically from the HTML page.       List<Game> games=new ArrayList<Game>();       // TODO Copy lines from next post here.       // Analysis by category. Ignore zeros.       double[][]areaJudgeMeans=new double[3][5];       double[][]areaJudgeVars=new double[3][5];       for (int area=0;area<3;area++){          for (int judge=0;judge<5;judge++){             int t=0,t2=0,n=0;             for (Game game:games){                int val=game.scores[area][judge];                if (val!=0){t+=val;t2+=val*val;n++;}             }             double mu=t/(double)n;             double var=(t2/(double)n)-mu*mu;             areaJudgeMeans[area][judge]=mu;             areaJudgeVars[area][judge]=var;          }       }       // Original values:       //printResults(games,new SimpleMean(),0);       // Current values:       //printResults(games,new MeanIgnoringZeros(),0);       // Fairest algorithm, assuming there should be no penalty for scoring 0s?       printResults(games,new NormalisedMeansIgnoringZeros(areaJudgeMeans[0],areaJudgeVars[0]),0);    }    private static void printResults(List<Game> games, ScoreAlgorithm alg, int area){       Collections.sort(games,new GameComparator(alg,area));       int longestName=0;       for (Game game:games) if (game.name.length()>longestName) longestName=game.name.length();       System.out.println("Sorting category "+areaNames[area]+" by "+alg);       int prev=Integer.MAX_VALUE,i=0;       for (Game game:games){          i++;          int score=alg.score(game,area);          String pos=score==prev?"= ":(i+".");          System.out.print(pad(3-pos.length())+pos+" "+game.name+pad(longestName+4-game.name.length()));          String sc=(score/10)+"."+(score%10);          System.out.println(pad(5-sc.length())+sc);          prev=score;       }    }    private static String pad(int n) {       char[] foo=new char[n];       Arrays.fill(foo,' ');       return new String(foo);    }    private static class Game {       public String name;       public int[][] scores=new int[3][5];       public Game(String name,int[][] scores){          this.name=name;          this.scores=scores;       }    }    public static interface ScoreAlgorithm {       public abstract int score(Game game, int area);    }    public static class GameComparator implements Comparator<Game> {       private final int area;       private final ScoreAlgorithm alg;       public GameComparator(ScoreAlgorithm alg,int area) {          this.alg=alg;          this.area=area;       }       public int compare(Game g1,Game g2) {return alg.score(g2,area)-alg.score(g1,area);}    }    public static class SimpleMean implements ScoreAlgorithm {       public int score(Game game,int area) {          int total=0;          for (int i=0;i<5;i++) total+=game.scores[area][i];          return (total*10+2)/5;       }       public String toString() {return "Simple mean";}    }    public static class MeanIgnoringZeros implements ScoreAlgorithm {       public int score(Game game,int area){          int total=0,n=0;          for (int i=0;i<5;i++) if (game.scores[area][i]!=0) {total+=game.scores[area][i];n++;}          return (total*10+n/2)/n;       }       public String toString() {return "Mean ignoring zero values";}    }    // Normalise to mean 70, variance 250.    public static class NormalisedMeansIgnoringZeros implements ScoreAlgorithm {       private final double[] means;       private final double[] vars;       public NormalisedMeansIgnoringZeros(double[] means,double[] vars) {          this.means=means;          this.vars=vars;       }       public int score(Game game,int area) {          double total=0,n=0;          for (int i=0;i<5;i++) {             if (game.scores[area][i]!=0) {total+=70+(game.scores[area][i]-means[i])*250/vars[i];n++;}          }          return (int)(10*total/n+.5);       }       public String toString() {return "Normalised means ignoring zeros";}    } }

[kevglass]

The fact that you don't know which API or framework is a hint that it's not necessarily the coder's fault.

That may be a hint to you, but the end user doesn't give a rats ass do they. They click the link and it didn't work. The fact its an NPE is a hint it is coder error. All of that's beside the point now really isn't it, the contest is over, the results are finalized.

Kev

Games published by our own members! Check 'em out!

[pjt33]

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21   22   23   24   25   26   27   28   29   30   31   32   33   34   35   36   37   38   39   40   41   42   43   44   45   46   47   48   49   50   51   52   53   54   55   56   57   58   59   60   61   62   63   64   65   66   67

games.add(new Game("4bidden Fruit", new int[][]{{96,76,90,100,90},{97,70,90,86,90},{96,84,85,96,90}}));       games.add(new Game("4bsolution", new int[][]{{91,65,70,80,90},{98,60,75,80,85},{98,80,70,99,80}}));       games.add(new Game("4K Maze", new int[][]{{30,50,20,72,70},{15,55,20,74,70},{10,50,20,1,60}}));       games.add(new Game("4x4k", new int[][]{{67,80,70,82,92},{83,85,75,90,85},{70,85,70,90,80}}));       games.add(new Game("Alienz 4K", new int[][]{{53,82,55,92,70},{63,75,55,75,65},{60,85,55,75,70}}));       games.add(new Game("Anyone 4 Tennis?", new int[][]{{60,65,70,62,70},{62,82,75,84,80},{75,80,65,79,65}}));       games.add(new Game("Arena4k", new int[][]{{69,62,60,75,50},{62,73,60,60,55},{55,70,60,40,50}}));       games.add(new Game("Asteroid Alert!", new int[][]{{59,82,80,87,75},{78,92,80,94,65},{60,87,75,94,75}}));       games.add(new Game("Beez Helper", new int[][]{{45,60,40,86,70},{50,73,45,75,65},{55,64,55,89,70}}));       games.add(new Game("Bio 4K", new int[][]{{54,93,70,68,70},{50,97,60,72,65},{60,90,60,81,65}}));       games.add(new Game("BlisterBall", new int[][]{{62,40,80,62,60},{50,50,75,45,65},{40,40,75,15,50}}));       games.add(new Game("BlockBuster4k", new int[][]{{48,92,80,92,85},{46,85,70,86,85},{55,82,80,93,85}}));       games.add(new Game("Bridge4k", new int[][]{{92,90,0,92,90},{92,92,0,100,95},{70,80,0,89,85}}));       games.add(new Game("Coffee Shop Puzzle", new int[][]{{78,45,75,72,65},{68,75,70,86,65},{84,83,80,89,75}}));       games.add(new Game("Conquest of Planets", new int[][]{{92,82,75,82,80},{80,88,80,71,80},{80,78,70,68,75}}));       games.add(new Game("Crack Tower Defense", new int[][]{{65,69,75,62,70},{40,79,75,78,75},{30,74,70,45,65}}));       games.add(new Game("Crimsonland4k version 1.001", new int[][]{{67,58,70,86,0},{55,50,80,82,0},{48,50,65,82,0}}));       games.add(new Game("Critters4k", new int[][]{{84,72,85,89,70},{95,78,75,90,75},{60,68,75,55,70}}));       games.add(new Game("Deathchase", new int[][]{{60,57,70,70,60},{65,70,75,65,60},{70,80,65,70,60}}));       games.add(new Game("Desert Bus", new int[][]{{65,65,20,5,50},{55,75,10,30,50},{55,72,40,50,50}}));       games.add(new Game("Doodles", new int[][]{{72,85,70,78,60},{75,94,80,89,75},{62,83,65,80,55}}));       games.add(new Game("Dragon 4k Boxing", new int[][]{{18,73,60,70,65},{25,70,70,80,65},{38,82,60,90,65}}));       games.add(new Game("F-Zero 4K", new int[][]{{66,90,75,66,85},{67,90,80,84,75},{78,85,75,87,90}}));       games.add(new Game("Fortress4K", new int[][]{{94,92,70,81,90},{88,95,70,86,85},{82,92,75,89,80}}));       games.add(new Game("Frequent Flier", new int[][]{{25,78,0,68,0},{45,88,0,91,0},{25,75,0,42,0}}));       games.add(new Game("Grasshopper4k", new int[][]{{68,82,60,60,80},{70,89,60,78,80},{75,87,65,88,85}}));       games.add(new Game("Gravitational Fourks", new int[][]{{70,66,0,65,0},{73,50,0,68,0},{65,70,0,61,0}}));       games.add(new Game("HexoDama", new int[][]{{72,86,80,45,70},{83,85,75,76,75},{64,80,75,74,65}}));       games.add(new Game("Honey4kKeeper", new int[][]{{42,83,65,81,75},{40,86,65,73,70},{45,90,65,89,75}}));       games.add(new Game("Hunt 4k Bread", new int[][]{{82,82,80,91,95},{90,91,70,88,95},{68,87,75,90,90}}));       games.add(new Game("i4kopter", new int[][]{{56,63,65,40,65},{58,68,65,70,70},{42,72,60,55,60}}));       games.add(new Game("Ice Fighters", new int[][]{{92,86,70,80,75},{98,90,75,89,75},{92,93,75,82,75}}));       games.add(new Game("Infinite Platformer 4k", new int[][]{{48,0,70,74,60},{30,0,75,72,55},{38,0,75,70,60}}));       games.add(new Game("Invaders", new int[][]{{94,85,75,85,75},{85,96,75,82,75},{85,92,70,79,70}}));       games.add(new Game("J-Type", new int[][]{{86,65,75,88,80},{78,60,70,87,75},{72,60,70,86,70}}));       games.add(new Game("Jetp4k", new int[][]{{75,85,75,80,80},{68,90,75,75,70},{76,88,75,85,75}}));       games.add(new Game("jm00 - a boomshine clone in 4k v1.2", new int[][]{{86,82,80,82,92},{74,90,75,92,85},{72,75,75,88,100}}));       games.add(new Game("Just Get Bigger 4K", new int[][]{{85,87,80,68,90},{78,95,75,80,85},{90,90,80,90,90}}));       games.add(new Game("Kart", new int[][]{{80,75,80,82,75},{82,86,80,80,80},{82,82,80,80,75}}));       games.add(new Game("Keggle", new int[][]{{91,90,80,78,90},{90,95,80,86,90},{82,85,75,89,90}}));       games.add(new Game("Left 4k Dead", new int[][]{{100,97,80,99,95},{100,95,75,99,95},{97,93,75,100,90}}));       games.add(new Game("MEG4kMAN", new int[][]{{95,95,75,86,70},{97,98,80,92,70},{95,95,80,92,65}}));       games.add(new Game("NiGHTS 4k", new int[][]{{90,84,0,85,80},{90,88,0,87,75},{95,87,0,100,80}}));       games.add(new Game("PacPitfall4k", new int[][]{{65,0,65,81,60},{80,0,60,92,60},{70,0,65,86,55}}));       games.add(new Game("Pixeloids4k", new int[][]{{92,89,0,91,90},{92,88,0,100,95},{88,82,0,94,75}}));       games.add(new Game("Putty Shuffle", new int[][]{{93,92,75,97,75},{85,90,75,86,80},{85,94,70,85,70}}));       games.add(new Game("Q*Bert4K", new int[][]{{80,0,80,82,80},{75,0,75,86,75},{70,0,75,94,85}}));       games.add(new Game("QuadSquad4k", new int[][]{{55,50,0,65,65},{40,60,0,65,70},{45,45,0,65,65}}));       games.add(new Game("Red Baron 4K", new int[][]{{85,96,70,98,0},{87,95,75,81,0},{92,100,75,96,0}}));       games.add(new Game("RevolvoMan-4K", new int[][]{{94,88,80,81,70},{92,92,75,84,70},{92,92,80,74,70}}));       games.add(new Game("RhythmSphere", new int[][]{{60,73,75,83,75},{75,76,75,89,80},{65,69,70,62,75}}));       games.add(new Game("Robo rampage", new int[][]{{82,93,75,93,85},{80,92,75,84,75},{82,90,74,96,90}}));       games.add(new Game("Run Over Zombies", new int[][]{{70,83,70,92,75},{85,90,75,99,75},{85,87,65,95,75}}));       games.add(new Game("Scr4mble", new int[][]{{55,60,60,52,70},{70,60,65,73,70},{50,60,75,72,70}}));       games.add(new Game("Sea Spin", new int[][]{{78,76,75,81,75},{78,86,75,77,70},{80,84,70,88,85}}));       games.add(new Game("Sea through", new int[][]{{45,73,75,78,85},{60,87,75,96,95},{40,40,70,84,80}}));       games.add(new Game("Space Paranoids 2D4K", new int[][]{{82,86,75,70,80},{90,90,70,87,75},{78,94,75,87,80}}));       games.add(new Game("Splosh4k", new int[][]{{90,73,75,87,95},{85,70,75,90,90},{82,65,70,96,95}}));       games.add(new Game("Star 4x", new int[][]{{60,88,0,61,70},{84,93,0,91,75},{60,80,0,85,60}}));       games.add(new Game("submarin", new int[][]{{40,84,75,66,80},{65,89,70,89,75},{25,95,75,76,85}}));       games.add(new Game("Super Marble World", new int[][]{{76,70,75,92,75},{85,75,75,90,70},{68,65,70,92,80}}));       games.add(new Game("Tekicars4k", new int[][]{{79,90,80,79,85},{82,92,75,74,85},{72,88,75,72,80}}));       games.add(new Game("Treasure4k", new int[][]{{78,70,75,82,70},{72,82,75,72,70},{92,78,70,87,65}}));       games.add(new Game("Ultimate Tic-Tac-Toe", new int[][]{{75,60,60,50,85},{75,60,70,65,90},{84,60,60,85,85}}));       games.add(new Game("Virtual On 4k", new int[][]{{50,41,60,68,55},{72,40,70,79,60},{45,35,60,52,50}}));       games.add(new Game("Word Twister", new int[][]{{56,83,70,84,80},{75,60,60,68,70},{68,74,75,82,80}}));       games.add(new Game("World Rally Driver 4k", new int[][]{{82,90,60,91,95},{82,93,60,89,95},{84,90,50,86,95}}));

[pjt33]

That may be a hint to you, but the end user doesn't give a rats ass do they. They click the link and it didn't work. The fact its an NPE is a hint it is coder error. All of that's beside the point now really isn't it, the contest is over, the results are finalized.

Did you get an NPE with Gravitational Fourks or are you confusing me with someone else?

[kevglass]

I'm not confusing anything, I wrote and you quoted:

One could also argue that it's a coding contest and failing with an NPE is a failure to code to some API or framework given how many other games managed to run on the same setup.

If your game didn't get a zero for failing with an NPE the comment above doesn't really apply to it does it?

Kev

[pjt33]

I'm not confusing anything, I wrote and you quoted:
If your game didn't get a zero for failing with an NPE the comment above doesn't really apply to it does it?

My game got two zeros. Versions of Java not specified, but with 1.6.0_12 I get

1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18   19   20   21

java.lang.InternalError: **************************************************************** ERROR: the javaplugin.version system property wasn't picked up by the com.sun.deploy.Environment class. This probably happened because of a change to the initialization order in PluginMain where the deployment classes are being initialized too early. This will break jar cache versioning, and possibly other things. Please undo your recent changes and rethink them. ****************************************************************    at sun.plugin2.applet.Applet2Environment.initialize(Applet2Environment.java:113)    at sun.plugin2.applet.viewer.JNLP2Viewer.run(JNLP2Viewer.java:195)    at sun.plugin2.applet.viewer.JNLP2Viewer.main(JNLP2Viewer.java:63)    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)    at java.lang.reflect.Method.invoke(Method.java:597)    at com.sun.javaws.Launcher.executeApplet(Launcher.java:1388)    at com.sun.javaws.Launcher.executeMainClass(Launcher.java:1246)    at com.sun.javaws.Launcher.doLaunchApp(Launcher.java:1066)    at com.sun.javaws.Launcher.run(Launcher.java:116)    at java.lang.Thread.run(Thread.java:619)

No mention of any NPEs.

[kevglass]

So then my comment wouldn't apply to your game then. Why quote it? Since the results have now been updated to ignore zero anyway, whats to worry about?

Kev

[Ranger]

Wow!  Excellent Judging!  Very helpful reviews.  Big thanks to Appel, Chris, and Demonpants for the great comments!  Darkfrog, errr, thanks for playing it! 

Appel, the Java4K Result page looks great!  Excellent job!

Congrats to Markus for a well deserved victory.  Gloat away!

[ShannonSmith]

Congrats Markus! But you really should have left it till the last week to submit, you probably put off a number of potential entrants.

Anyone else think percentage scores are unnecessarily precise given that the scores from each judge differed by over 50% in some cases. I think out of ten scores would probably be better for future contests.

Games published by our own members! Check 'em out!

[Morre]

Regarding "remove lowest score for everyone":

I'm fine with this. An ever better way might be to remove the lowest and highest score for each game (just taking an average between the middle 3). I believe this would be a very fair way to do it (a Truncated mean as suggested by oNyx).

In either case, I think the scores should be finalized very shortly (I'm fine with whatever method the organizers decide to use), and then a plan set up for next year. I propose truncated average (remove lowest and highest score).


EDIT: On a side note, I believe the NPE discussed might be the one that mentioned in a comment by darkfrog for one of my games. Whether it's because of faulty code on my behalf or problems with obfuscation or differences in the JVM is hard to tell. I might see if I can get some other Vista 64 user to run it and see if I can pinpoint it.

[Ranger]

Regarding "remove lowest score for everyone":

I'm fine with this. An ever better way might be to remove the lowest and highest score for each game (just taking an average between the middle 3). I believe this would be a very fair way to do it (a Truncated mean as suggested by oNyx).

In either case, I think the scores should be finalized very shortly (I'm fine with whatever method the organizers decide to use), and then a plan set up for next year. I propose truncated average (remove lowest and highest score).

Excellent idea!

[appel]

Although all these ideas are worth chatting about, I do not think applying them to the current results will help (the competition). Cooking more with the results will only do harm.

We will instead try to apply these ideas to next year's judging. Which reminds me, we should probably discuss the overall judging process instead of just how to calculate the percentages. This is probably a topic for another topic.

[SimonH]

The score chart has now been corrected!

Noooo! Now 4x4k is 30th! This can't be right! 4x4K is much better than Jetp4k (now 26th)!
No offence to the judges but - WE NEED A BETTER METHOD!

EDIT: I'll tone this down later *hic* but s'true! *hic* *urp*

[appel]

Noooo! Now 4x4k is 30th! This can't be right! 4x4K is much better than Jetp4k (now 26th)!
No offence to the judges but - WE NEED A BETTER METHOD!

IDEA ALERT

I've been thinking about the judging process and about this problem of "this game is better than that game" and using percentages to solve that. Doesn't seem like the best way.

IMO we should just have, let's say five bins, and allow the judges to put the games into in order to rate them, these could be:

Bin 1: SUPERB game
Bin 2: Excellent game
Bin 3: Good fun
Bin 4: Alright
Bin 5: Moving on

And within each bin, the judges could sort the games in the this-game-is-better-than-this-one order.

This could deliver much more meaningful results, as games are sorted better, and the daunting percentages are removed which actually hold no real meaning.

This also makes the judging process easier, as it's much more easier to put many games into just a handful of categories, and then deal with the games in each category.

[CyanPrime]

Virtual On 4K sucks T_______T
DeamonPants, you seem like you'd like Blue Fiend's new controls. You should re-check it out.

[Ranger]

IDEA ALERT

I've been thinking about the judging process and about this problem of "this game is better than that game" and using percentages to solve that. Doesn't seem like the best way.

IMO we should just have, let's say five bins, and allow the judges to put the games into in order to rate them, these could be:

Bin 1: SUPERB game
Bin 2: Excellent game
Bin 3: Good fun
Bin 4: Alright
Bin 5: Moving on

And within each bin, the judges could sort the games in the this-game-is-better-than-this-one order.

This could deliver much more meaningful results, as games are sorted better, and the daunting percentages are removed which actually hold no real meaning.

This also makes the judging process easier, as it's much more easier to put many games into just a handful of categories, and then deal with the games in each category.

I like it!  And it would be cool if users could also use this method to judge (but keep their score separate to the real judges scores).

[Morre]

With the new method proposed, I feel the need to put n/a ("Couldn't run" or the like) is important. Apart from that... Yeah, I'm fine with the idea of having more of a bins sort of rating (1-5 or 1-10 per game).

I'm not sure there's much point sorting the entries within each bin if we're talking community vote - that should sort itself out with more than say, 10 or 15 votes. As for judges... that's trickier.

A method used rather successfully in the Ludumdare contest is to present each participant with a random games order judging page. People will generally start from the top, so if everybody rates say, a third of the games, each game will still get a decent amount of votes. There are problems with this method as well, but it's worked reasonably well for them I'd say. In order to be successful, however, it needs to be coupled with a strong encouragement for each participant to judge a few entries.

EDIT: Also, I feel like an idiot for having brought this whole 0-points issue up. Sorry about the mess! I've been coming out as all negative here, when all I really wanted was to point out the issue. I would've been entirely fine with the results table staying the same and new scoring mechanisms being used next year. With the new scoring table, two of my four games have an unfair advantage because darkfrog generally gave lower scores... :S This sort of problem is the reason we should remove lowest and highest scores for all games or normalize the judges' results.

I'm feeling really guilty right now.

[pjt33]

So then my comment wouldn't apply to your game then. Why quote it?

It was well after my bedtime and I wasn't expressing myself as clearly as I would like. My point was that, leaving aside the issue of which exception is thrown, we're targetting not one API/framework but several dozen, and we can't test on all of them. Lurking in the back of my mind also was the fact that in efforts to save a few bytes some people are straying into areas which the spec doesn't cover clearly. For example, AFAIK the spec for Applet.getGraphics() says nothing about it returning null at some stages in the applet's lifecycle, but some people found that that was the case with a small number of VMs.

Since the results have now been updated to ignore zero anyway, whats to worry about?

Next year, obviously.

Anyone else think percentage scores are unnecessarily precise given that the scores from each judge differed by over 50% in some cases. I think out of ten scores would probably be better for future contests.

darkfrog's standard deviation for presentation was 10.3, so with less precision nearly everyone would have been lumped together in three buckets under that scheme. In general most people scarcely use the bottom half of a 1-10 scale. I was thinking about something similar to appel's buckets suggestion, although that needs work to get something quantifiable which can be averaged.

Noooo! Now 4x4k is 30th! This can't be right! 4x4K is much better than Jetp4k (now 26th)!

In general there seems to be a moderate "nostalgia bonus" for basing it on a game the judges played in their youth.

EDIT: Also, I feel like an idiot for having brought this whole 0-points issue up. <snip>
I'm feeling really guilty right now.

Someone else would have brought it up. Relax.

[kevglass]

It was well after my bedtime and I wasn't expressing myself as clearly as I would like.

Fair enough.

Next year, obviously.

Seems like that's already being thought about, so thats great too.

I won't be getting involved in the contest again. This sort of stuff afterwards just leaves what was fun activity with a nasty taste. The only good option is community voting and reviewing.

Kev

[cylab]

I won't be getting involved in the contest again. This sort of stuff afterwards just leaves what was fun activity with a nasty taste.

Come on, that's late aprilsfool, isn't it. It's just natural to discuss the judging (and possible problematic scores) and I don't think this makes the excelent games this year any worse.

The only good option is community voting and reviewing.

Hmm, not sure about this. Should first be tried parallel to judge votes to see, if there are enough community votes/reviews for all games.

[woogley]

This sort of stuff afterwards just leaves what was fun activity with a nasty taste.

Yeah, I agree. This quote always reminds me why I quit hosting:

lol, no good deed goes unpunished woogley

Thanks appel! The torch is yours!!

Congrats to all game devs who wrote excellent games and kept the 4K fun. For those who bicker about pointless stats, please take that shit to the Flash 4K contest or something. It's not that big of a deal.

[Markus_Persson]

I think having (from next year on) the judges simply list the games from best to.. uhm.. least best is actually a very good idea. It removes all subjective scoring from the equation and enforces a uniform point system. Having bins as well has the added benefit of distinguishing games into discrete groups, so there could be two AWESOME, a hundred VERY GOOD, ten OK, and two NOT OK.

I don't mind these discussion about scoring, but I'm very happy apple said there would be no more fiddling.

And don't quit, kev.. don't be like that.

[edit:]
It seems like mojang.com is down.. I can't find out why until I get home tonight. Did we get slashdotted or something?

[edit edit:]
Nevermind..

[sunsett]

?! It probably didn't affect first place, but that's a much weaker statement than not affecting the outcome.

BTW darkfrog, could you say which version of Java you were using?

I don't remember which machine I was running yours on.  It was either Java 1.6u7 or Java 1.6u12, but I'm not sure. However, since 1.6u7 is standard on Mac and 1.6u12 is the latest version I think that though it sucks for you, it is fair to expect the game should run without issue.  Remember that we are judging a game that was coded and part of the coding is compatibility.  If you haven't tested your game to run properly on 1.6 or greater then you have to lose points.  It sucks that it drops you to zero, but like Chris said, what other score can I give if I can't play it?

[sunsett]

Wow!  Excellent Judging!  Very helpful reviews.  Big thanks to Appel, Chris, and Demonpants for the great comments!  Darkfrog, errr, thanks for playing it! 

Appel, the Java4K Result page looks great!  Excellent job!

Congrats to Markus for a well deserved victory.  Gloat away!

What game was yours? Was yours the one I didn't put any comment at all in?

[pjt33]

I don't remember which machine I was running yours on.  It was either Java 1.6u7 or Java 1.6u12, but I'm not sure. However, since 1.6u7 is standard on Mac and 1.6u12 is the latest version I think that though it sucks for you, it is fair to expect the game should run without issue.  Remember that we are judging a game that was coded and part of the coding is compatibility.  If you haven't tested your game to run properly on 1.6 or greater then you have to lose points.  It sucks that it drops you to zero, but like Chris said, what other score can I give if I can't play it?

It does run properly on the versions of 1.6 which were released when I finished it (i.e. up to 1.6u11). u12 came out in the last week of February, and u13 sometime in March, it seems, although I didn't know it existed until today. The breakage in u12 looks like a bug in Webstart, and u13 seems to have a different bug in Webstart based on the stack trace a friend sent me today.

P.S. Surely the latest version is 1.7?

[Orangy Tang]

I don't remember which machine I was running yours on.  It was either Java 1.6u7 or Java 1.6u12, but I'm not sure.

Interesting, was that what you used for NiGHTS and had problems with as well? Because I tested with 1.6 (and it's working on 1.6.0_07 32-bit WinXP here), so it sounds like there's something else going on.

[sunsett]

EDIT: Also, I feel like an idiot for having brought this whole 0-points issue up. Sorry about the mess! I've been coming out as all negative here, when all I really wanted was to point out the issue. I would've been entirely fine with the results table staying the same and new scoring mechanisms being used next year. With the new scoring table, two of my four games have an unfair advantage because darkfrog generally gave lower scores... :S This sort of problem is the reason we should remove lowest and highest scores for all games or normalize the judges' results.

I did in general give lower scores, but it would because I kept holding out for a game that stood out as exceptional to give high marks to.  Last year I marked everything relatively high and then near the end found a game that complete changed the standard and had to go back and move everything else down to differentiate it.  Please don't take that to mean the games were of bad quality, but rather for the most part most of them were pretty high quality, and though I didn't give 100s to anyone, the reason the score differentiation for me wasn't very great for most was because most of the games were of a generally high quality.  However, the fact that I scored everything relatively lower it shouldn't have any impact on the order of results, just the end scores.

[sunsett]

It does run properly on the versions of 1.6 which were released when I finished it (i.e. up to 1.6u11). u12 came out in the last week of February, and u13 sometime in March, it seems, although I didn't know it existed until today. The breakage in u12 looks like a bug in Webstart, and u13 seems to have a different bug in Webstart based on the stack trace a friend sent me today.

P.S. Surely the latest version is 1.7?

Latest stable version of Java is 1.6u12 and though it sucks to have this be an issue you have to be able to support the currently stable version of Java.  I would be curious to know what it is in your code that is causing a breakage in Java in u12 though, that's very odd.

[sunsett]

Interesting, was that what you used for NiGHTS and had problems with as well? Because I tested with 1.6 (and it's working on 1.6.0_07 32-bit WinXP here), so it sounds like there's something else going on.

I did about half my testing on a Mac with 1.6u7 and the other half on Windows Vista 64-bit with 1.6u12.

Anyone that I gave a zero score for failing to run the application please feel free to PM me if you'd like me to help you resolve the issue.  I would also just like to be able to play your games.