Java-Gaming.org Hi !
Featured games (83)
games approved by the League of Dukes
Games in Showcase (538)
Games in Android Showcase (132)
games submitted by our members
Games in WIP (600)
games currently in development
News: Read the Java Gaming Resources, or peek at the official Java tutorials
 
    Home     Help   Search   Login   Register   
Pages: 1 [2] 3
  ignore  |  Print  
  Judging Results  (Read 23780 times)
0 Members and 1 Guest are viewing this topic.
Offline pjt33
« Reply #30 - Posted 2009-04-01 23:12:20 »

Just do it (TM).

However you should probably also put up the following variants:

- Remove all the highest scores for each entry
- Remove all the lowest scores for each entry
- Remove all both the highest and the lowest for each entry
- Remove a particular judge's results for each entry
- Make all the scores for every game but the one selected zero (ok, maybe not this one Wink).

It'd hardly be fair to judge some entries on 4 judge's results and some on 5 given the disparity in the judges ranges of scores and approach to judging.

This could go on and on and on...
The way to handle the disparity in ranges of scores is to normalise. Doing that and discounting 0s gives:
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
Sorting category Overall by Normalised means ignoring zeros
 1. Left 4k Dead                            94.5
 2. 4bidden Fruit                           90.1
 3. Bridge4k                                88.8
 4. Pixeloids4k                             88.2
 5. Hunt 4k Bread                           86.4
 6. Keggle                                  85.1
 7. Robo rampage                            84.1
 8. jm00 - a boomshine clone in 4k v1.2     84.0
 =  Red Baron 4K                            84.0
10. Fortress4K                              83.5
11. Splosh4k                                82.9
12. Putty Shuffle                           82.3
13. World Rally Driver 4k                   82.0
14. Tekicars4k                              81.6
15. Just Get Bigger 4K                      81.4
16. BlockBuster4k                           80.3
17. NiGHTS 4k                               79.8
18. MEG4kMAN                                79.1
19. Q*Bert4K                                78.7
20. Conquest of Planets                     78.5
21. Invaders                                78.2
22. RevolvoMan-4K                           77.8
23. 4x4k                                    77.5
24. Jetp4k                                  76.4
25. 4bsolution                              75.9
26. Space Paranoids 2D4K                    75.6
27. Critters4k                              75.5
 =  Ice Fighters                            75.5
29. F-Zero 4K                               75.4
30. Kart                                    74.6
31. J-Type                                  74.5
32. Asteroid Alert!                         74.4
33. Run Over Zombies                        74.0
34. Super Marble World                      73.2
35. Sea Spin                                72.8
36. Word Twister                            72.3
37. Sea through                             70.5
38. RhythmSphere                            69.8
39. Treasure4k                              69.5
40. submarin                                68.1
41. Bio 4K                                  67.3
42. HexoDama                                66.6
43. Honey4kKeeper                           66.1
44. Doodles                                 66.0
45. Grasshopper4k                           65.6
46. Crimsonland4k version 1.001             65.5
47. Alienz 4K                               64.5
 =  Star 4x                                 64.5
49. Crack Tower Defense                     63.2
50. Gravitational Fourks                    61.2
51. Ultimate Tic-Tac-Toe                    60.9
52. Anyone 4 Tennis?                        59.8
 =  PacPitfall4k                            59.8
54. Coffee Shop Puzzle                      59.0
55. Infinite Platformer 4k                  57.0
56. Frequent Flier                          56.4
57. Deathchase                              55.5
58. BlisterBall                             53.0
59. Scr4mble                                52.5
60. Dragon 4k Boxing                        52.2
61. Arena4k                                 51.9
62. Beez Helper                             51.6
63. i4kopter                                50.6
64. QuadSquad4k                             50.1
65. Virtual On 4k                           44.3
66. 4K Maze                                 37.5
67. Desert Bus                              24.7


Code to follow, because it doesn't fit in one post.

Quote
[size=5pt]One could also argue that it's a coding contest and failing with an NPE is a failure to code to some API or framework given how many other games managed to run on the same setup.[/size]
The fact that you don't know which API or framework is a hint that it's not necessarily the coder's fault. My game broke with a version of Java which was released after I finished writing it. I don't know which API or framework broke, nor in what way, because Webstart is rather sparse with its error messages. The only error message it shows seems to be an internal one.
Offline pjt33
« Reply #31 - Posted 2009-04-01 23:20:47 »

Had to split it because of message length restrictions. See the TODO.
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
68  
69  
70  
71  
72  
73  
74  
75  
76  
77  
78  
79  
80  
81  
82  
83  
84  
85  
86  
87  
88  
89  
90  
91  
92  
93  
94  
95  
96  
97  
98  
99  
100  
101  
102  
103  
104  
105  
106  
107  
108  
109  
110  
111  
112  
113  
114  
115  
116  
117  
118  
119  
120  
121  
122  
123  
124  
125  
126  
127  
import java.util.*;
public class Analyse4k{
   private static String[] areaNames={"Overall","Technical","Presentation"};
   public static void main(String[] args){
      // Note: values extracted automatically from the HTML page.
      List<Game> games=new ArrayList<Game>();
      // TODO Copy lines from next post here.

      // Analysis by category. Ignore zeros.
      double[][]areaJudgeMeans=new double[3][5];
      double[][]areaJudgeVars=new double[3][5];
      for (int area=0;area<3;area++){
         for (int judge=0;judge<5;judge++){
            int t=0,t2=0,n=0;
            for (Game game:games){
               int val=game.scores[area][judge];
               if (val!=0){t+=val;t2+=val*val;n++;}
            }
            double mu=t/(double)n;
            double var=(t2/(double)n)-mu*mu;
            areaJudgeMeans[area][judge]=mu;
            areaJudgeVars[area][judge]=var;
         }
      }

      // Original values:
      //printResults(games,new SimpleMean(),0);

      // Current values:
      //printResults(games,new MeanIgnoringZeros(),0);

      // Fairest algorithm, assuming there should be no penalty for scoring 0s?
      printResults(games,new NormalisedMeansIgnoringZeros(areaJudgeMeans[0],areaJudgeVars[0]),0);
   }

   private static void printResults(List<Game> games, ScoreAlgorithm alg, int area){
      Collections.sort(games,new GameComparator(alg,area));
      int longestName=0;
      for (Game game:games) if (game.name.length()>longestName) longestName=game.name.length();

      System.out.println("Sorting category "+areaNames[area]+" by "+alg);
      int prev=Integer.MAX_VALUE,i=0;
      for (Game game:games){
         i++;
         int score=alg.score(game,area);
         String pos=score==prev?"= ":(i+".");
         System.out.print(pad(3-pos.length())+pos+" "+game.name+pad(longestName+4-game.name.length()));
         String sc=(score/10)+"."+(score%10);
         System.out.println(pad(5-sc.length())+sc);
         prev=score;
      }
   }

   private static String pad(int n) {
      char[] foo=new char[n];
      Arrays.fill(foo,' ');
      return new String(foo);
   }

   private static class Game {
      public String name;
      public int[][] scores=new int[3][5];

      public Game(String name,int[][] scores){
         this.name=name;
         this.scores=scores;
      }
   }

   public static interface ScoreAlgorithm {
      public abstract int score(Game game, int area);
   }

   public static class GameComparator implements Comparator<Game> {
      private final int area;
      private final ScoreAlgorithm alg;

      public GameComparator(ScoreAlgorithm alg,int area) {
         this.alg=alg;
         this.area=area;
      }

      public int compare(Game g1,Game g2) {return alg.score(g2,area)-alg.score(g1,area);}
   }

   public static class SimpleMean implements ScoreAlgorithm {
      public int score(Game game,int area) {
         int total=0;
         for (int i=0;i<5;i++) total+=game.scores[area][i];
         return (total*10+2)/5;
      }

      public String toString() {return "Simple mean";}
   }

   public static class MeanIgnoringZeros implements ScoreAlgorithm {
      public int score(Game game,int area){
         int total=0,n=0;
         for (int i=0;i<5;i++) if (game.scores[area][i]!=0) {total+=game.scores[area][i];n++;}
         return (total*10+n/2)/n;
      }

      public String toString() {return "Mean ignoring zero values";}
   }

   // Normalise to mean 70, variance 250.
   public static class NormalisedMeansIgnoringZeros implements ScoreAlgorithm {
      private final double[] means;
      private final double[] vars;

      public NormalisedMeansIgnoringZeros(double[] means,double[] vars) {
         this.means=means;
         this.vars=vars;
      }

      public int score(Game game,int area) {
         double total=0,n=0;
         for (int i=0;i<5;i++) {
            if (game.scores[area][i]!=0) {total+=70+(game.scores[area][i]-means[i])*250/vars[i];n++;}
         }

         return (int)(10*total/n+.5);
      }

      public String toString() {return "Normalised means ignoring zeros";}
   }
}
Offline kevglass

« JGO Spiffy Duke »


Medals: 211
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #32 - Posted 2009-04-01 23:21:04 »

Quote
The fact that you don't know which API or framework is a hint that it's not necessarily the coder's fault.

That may be a hint to you, but the end user doesn't give a rats ass do they. They click the link and it didn't work. The fact its an NPE is a hint it is coder error. All of that's beside the point now really isn't it, the contest is over, the results are finalized.

Kev

Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline pjt33
« Reply #33 - Posted 2009-04-01 23:21:22 »

1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
22  
23  
24  
25  
26  
27  
28  
29  
30  
31  
32  
33  
34  
35  
36  
37  
38  
39  
40  
41  
42  
43  
44  
45  
46  
47  
48  
49  
50  
51  
52  
53  
54  
55  
56  
57  
58  
59  
60  
61  
62  
63  
64  
65  
66  
67  
      games.add(new Game("4bidden Fruit", new int[][]{{96,76,90,100,90},{97,70,90,86,90},{96,84,85,96,90}}));
      games.add(new Game("4bsolution", new int[][]{{91,65,70,80,90},{98,60,75,80,85},{98,80,70,99,80}}));
      games.add(new Game("4K Maze", new int[][]{{30,50,20,72,70},{15,55,20,74,70},{10,50,20,1,60}}));
      games.add(new Game("4x4k", new int[][]{{67,80,70,82,92},{83,85,75,90,85},{70,85,70,90,80}}));
      games.add(new Game("Alienz 4K", new int[][]{{53,82,55,92,70},{63,75,55,75,65},{60,85,55,75,70}}));
      games.add(new Game("Anyone 4 Tennis?", new int[][]{{60,65,70,62,70},{62,82,75,84,80},{75,80,65,79,65}}));
      games.add(new Game("Arena4k", new int[][]{{69,62,60,75,50},{62,73,60,60,55},{55,70,60,40,50}}));
      games.add(new Game("Asteroid Alert!", new int[][]{{59,82,80,87,75},{78,92,80,94,65},{60,87,75,94,75}}));
      games.add(new Game("Beez Helper", new int[][]{{45,60,40,86,70},{50,73,45,75,65},{55,64,55,89,70}}));
      games.add(new Game("Bio 4K", new int[][]{{54,93,70,68,70},{50,97,60,72,65},{60,90,60,81,65}}));
      games.add(new Game("BlisterBall", new int[][]{{62,40,80,62,60},{50,50,75,45,65},{40,40,75,15,50}}));
      games.add(new Game("BlockBuster4k", new int[][]{{48,92,80,92,85},{46,85,70,86,85},{55,82,80,93,85}}));
      games.add(new Game("Bridge4k", new int[][]{{92,90,0,92,90},{92,92,0,100,95},{70,80,0,89,85}}));
      games.add(new Game("Coffee Shop Puzzle", new int[][]{{78,45,75,72,65},{68,75,70,86,65},{84,83,80,89,75}}));
      games.add(new Game("Conquest of Planets", new int[][]{{92,82,75,82,80},{80,88,80,71,80},{80,78,70,68,75}}));
      games.add(new Game("Crack Tower Defense", new int[][]{{65,69,75,62,70},{40,79,75,78,75},{30,74,70,45,65}}));
      games.add(new Game("Crimsonland4k version 1.001", new int[][]{{67,58,70,86,0},{55,50,80,82,0},{48,50,65,82,0}}));
      games.add(new Game("Critters4k", new int[][]{{84,72,85,89,70},{95,78,75,90,75},{60,68,75,55,70}}));
      games.add(new Game("Deathchase", new int[][]{{60,57,70,70,60},{65,70,75,65,60},{70,80,65,70,60}}));
      games.add(new Game("Desert Bus", new int[][]{{65,65,20,5,50},{55,75,10,30,50},{55,72,40,50,50}}));
      games.add(new Game("Doodles", new int[][]{{72,85,70,78,60},{75,94,80,89,75},{62,83,65,80,55}}));
      games.add(new Game("Dragon 4k Boxing", new int[][]{{18,73,60,70,65},{25,70,70,80,65},{38,82,60,90,65}}));
      games.add(new Game("F-Zero 4K", new int[][]{{66,90,75,66,85},{67,90,80,84,75},{78,85,75,87,90}}));
      games.add(new Game("Fortress4K", new int[][]{{94,92,70,81,90},{88,95,70,86,85},{82,92,75,89,80}}));
      games.add(new Game("Frequent Flier", new int[][]{{25,78,0,68,0},{45,88,0,91,0},{25,75,0,42,0}}));
      games.add(new Game("Grasshopper4k", new int[][]{{68,82,60,60,80},{70,89,60,78,80},{75,87,65,88,85}}));
      games.add(new Game("Gravitational Fourks", new int[][]{{70,66,0,65,0},{73,50,0,68,0},{65,70,0,61,0}}));
      games.add(new Game("HexoDama", new int[][]{{72,86,80,45,70},{83,85,75,76,75},{64,80,75,74,65}}));
      games.add(new Game("Honey4kKeeper", new int[][]{{42,83,65,81,75},{40,86,65,73,70},{45,90,65,89,75}}));
      games.add(new Game("Hunt 4k Bread", new int[][]{{82,82,80,91,95},{90,91,70,88,95},{68,87,75,90,90}}));
      games.add(new Game("i4kopter", new int[][]{{56,63,65,40,65},{58,68,65,70,70},{42,72,60,55,60}}));
      games.add(new Game("Ice Fighters", new int[][]{{92,86,70,80,75},{98,90,75,89,75},{92,93,75,82,75}}));
      games.add(new Game("Infinite Platformer 4k", new int[][]{{48,0,70,74,60},{30,0,75,72,55},{38,0,75,70,60}}));
      games.add(new Game("Invaders", new int[][]{{94,85,75,85,75},{85,96,75,82,75},{85,92,70,79,70}}));
      games.add(new Game("J-Type", new int[][]{{86,65,75,88,80},{78,60,70,87,75},{72,60,70,86,70}}));
      games.add(new Game("Jetp4k", new int[][]{{75,85,75,80,80},{68,90,75,75,70},{76,88,75,85,75}}));
      games.add(new Game("jm00 - a boomshine clone in 4k v1.2", new int[][]{{86,82,80,82,92},{74,90,75,92,85},{72,75,75,88,100}}));
      games.add(new Game("Just Get Bigger 4K", new int[][]{{85,87,80,68,90},{78,95,75,80,85},{90,90,80,90,90}}));
      games.add(new Game("Kart", new int[][]{{80,75,80,82,75},{82,86,80,80,80},{82,82,80,80,75}}));
      games.add(new Game("Keggle", new int[][]{{91,90,80,78,90},{90,95,80,86,90},{82,85,75,89,90}}));
      games.add(new Game("Left 4k Dead", new int[][]{{100,97,80,99,95},{100,95,75,99,95},{97,93,75,100,90}}));
      games.add(new Game("MEG4kMAN", new int[][]{{95,95,75,86,70},{97,98,80,92,70},{95,95,80,92,65}}));
      games.add(new Game("NiGHTS 4k", new int[][]{{90,84,0,85,80},{90,88,0,87,75},{95,87,0,100,80}}));
      games.add(new Game("PacPitfall4k", new int[][]{{65,0,65,81,60},{80,0,60,92,60},{70,0,65,86,55}}));
      games.add(new Game("Pixeloids4k", new int[][]{{92,89,0,91,90},{92,88,0,100,95},{88,82,0,94,75}}));
      games.add(new Game("Putty Shuffle", new int[][]{{93,92,75,97,75},{85,90,75,86,80},{85,94,70,85,70}}));
      games.add(new Game("Q*Bert4K", new int[][]{{80,0,80,82,80},{75,0,75,86,75},{70,0,75,94,85}}));
      games.add(new Game("QuadSquad4k", new int[][]{{55,50,0,65,65},{40,60,0,65,70},{45,45,0,65,65}}));
      games.add(new Game("Red Baron 4K", new int[][]{{85,96,70,98,0},{87,95,75,81,0},{92,100,75,96,0}}));
      games.add(new Game("RevolvoMan-4K", new int[][]{{94,88,80,81,70},{92,92,75,84,70},{92,92,80,74,70}}));
      games.add(new Game("RhythmSphere", new int[][]{{60,73,75,83,75},{75,76,75,89,80},{65,69,70,62,75}}));
      games.add(new Game("Robo rampage", new int[][]{{82,93,75,93,85},{80,92,75,84,75},{82,90,74,96,90}}));
      games.add(new Game("Run Over Zombies", new int[][]{{70,83,70,92,75},{85,90,75,99,75},{85,87,65,95,75}}));
      games.add(new Game("Scr4mble", new int[][]{{55,60,60,52,70},{70,60,65,73,70},{50,60,75,72,70}}));
      games.add(new Game("Sea Spin", new int[][]{{78,76,75,81,75},{78,86,75,77,70},{80,84,70,88,85}}));
      games.add(new Game("Sea through", new int[][]{{45,73,75,78,85},{60,87,75,96,95},{40,40,70,84,80}}));
      games.add(new Game("Space Paranoids 2D4K", new int[][]{{82,86,75,70,80},{90,90,70,87,75},{78,94,75,87,80}}));
      games.add(new Game("Splosh4k", new int[][]{{90,73,75,87,95},{85,70,75,90,90},{82,65,70,96,95}}));
      games.add(new Game("Star 4x", new int[][]{{60,88,0,61,70},{84,93,0,91,75},{60,80,0,85,60}}));
      games.add(new Game("submarin", new int[][]{{40,84,75,66,80},{65,89,70,89,75},{25,95,75,76,85}}));
      games.add(new Game("Super Marble World", new int[][]{{76,70,75,92,75},{85,75,75,90,70},{68,65,70,92,80}}));
      games.add(new Game("Tekicars4k", new int[][]{{79,90,80,79,85},{82,92,75,74,85},{72,88,75,72,80}}));
      games.add(new Game("Treasure4k", new int[][]{{78,70,75,82,70},{72,82,75,72,70},{92,78,70,87,65}}));
      games.add(new Game("Ultimate Tic-Tac-Toe", new int[][]{{75,60,60,50,85},{75,60,70,65,90},{84,60,60,85,85}}));
      games.add(new Game("Virtual On 4k", new int[][]{{50,41,60,68,55},{72,40,70,79,60},{45,35,60,52,50}}));
      games.add(new Game("Word Twister", new int[][]{{56,83,70,84,80},{75,60,60,68,70},{68,74,75,82,80}}));
      games.add(new Game("World Rally Driver 4k", new int[][]{{82,90,60,91,95},{82,93,60,89,95},{84,90,50,86,95}}));
Offline pjt33
« Reply #34 - Posted 2009-04-01 23:22:58 »

That may be a hint to you, but the end user doesn't give a rats ass do they. They click the link and it didn't work. The fact its an NPE is a hint it is coder error. All of that's beside the point now really isn't it, the contest is over, the results are finalized.
Huh Did you get an NPE with Gravitational Fourks or are you confusing me with someone else?
Offline kevglass

« JGO Spiffy Duke »


Medals: 211
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #35 - Posted 2009-04-01 23:24:22 »

I'm not confusing anything, I wrote and you quoted:
Quote
One could also argue that it's a coding contest and failing with an NPE is a failure to code to some API or framework given how many other games managed to run on the same setup.

If your game didn't get a zero for failing with an NPE the comment above doesn't really apply to it does it?

Kev

Offline pjt33
« Reply #36 - Posted 2009-04-01 23:30:56 »

I'm not confusing anything, I wrote and you quoted:
If your game didn't get a zero for failing with an NPE the comment above doesn't really apply to it does it?
My game got two zeros. Versions of Java not specified, but with 1.6.0_12 I get
1  
2  
3  
4  
5  
6  
7  
8  
9  
10  
11  
12  
13  
14  
15  
16  
17  
18  
19  
20  
21  
java.lang.InternalError: 
****************************************************************
ERROR: the javaplugin.version system property wasn't picked up
by the com.sun.deploy.Environment class. This probably happened
because of a change to the initialization order in PluginMain
where the deployment classes are being initialized too early.
This will break jar cache versioning, and possibly other things.
Please undo your recent changes and rethink them.
****************************************************************
   at sun.plugin2.applet.Applet2Environment.initialize(Applet2Environment.java:113)
   at sun.plugin2.applet.viewer.JNLP2Viewer.run(JNLP2Viewer.java:195)
   at sun.plugin2.applet.viewer.JNLP2Viewer.main(JNLP2Viewer.java:63)
   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
   at java.lang.reflect.Method.invoke(Method.java:597)
   at com.sun.javaws.Launcher.executeApplet(Launcher.java:1388)
   at com.sun.javaws.Launcher.executeMainClass(Launcher.java:1246)
   at com.sun.javaws.Launcher.doLaunchApp(Launcher.java:1066)
   at com.sun.javaws.Launcher.run(Launcher.java:116)
   at java.lang.Thread.run(Thread.java:619)

No mention of any NPEs.
Offline kevglass

« JGO Spiffy Duke »


Medals: 211
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #37 - Posted 2009-04-01 23:33:40 »

So then my comment wouldn't apply to your game then. Why quote it? Since the results have now been updated to ignore zero anyway, whats to worry about?

Kev

Offline Ranger
« Reply #38 - Posted 2009-04-01 23:42:23 »

Wow!  Excellent Judging!  Very helpful reviews.  Big thanks to Appel, Chris, and Demonpants for the great comments!  Darkfrog, errr, thanks for playing it!  Smiley

Appel, the Java4K Result page looks great!  Excellent job!

Congrats to Markus for a well deserved victory.  Gloat away!
Offline ShannonSmith
« Reply #39 - Posted 2009-04-01 23:46:49 »

Congrats Markus! But you really should have left it till the last week to submit, you probably put off a number of potential entrants.

Anyone else think percentage scores are unnecessarily precise given that the scores from each judge differed by over 50% in some cases. I think out of ten scores would probably be better for future contests.
Games published by our own members! Check 'em out!
Legends of Yore - The Casual Retro Roguelike
Offline Morre

JGO Knight


Medals: 2
Projects: 10


I'm Dragonene on IRC.


« Reply #40 - Posted 2009-04-02 00:14:07 »

Regarding "remove lowest score for everyone":

I'm fine with this. An ever better way might be to remove the lowest and highest score for each game (just taking an average between the middle 3). I believe this would be a very fair way to do it (a Truncated mean as suggested by oNyx).

In either case, I think the scores should be finalized very shortly (I'm fine with whatever method the organizers decide to use), and then a plan set up for next year. I propose truncated average (remove lowest and highest score).


EDIT: On a side note, I believe the NPE discussed might be the one that mentioned in a comment by darkfrog for one of my games. Whether it's because of faulty code on my behalf or problems with obfuscation or differences in the JVM is hard to tell. I might see if I can get some other Vista 64 user to run it and see if I can pinpoint it.

Offline Ranger
« Reply #41 - Posted 2009-04-02 00:16:23 »

Regarding "remove lowest score for everyone":

I'm fine with this. An ever better way might be to remove the lowest and highest score for each game (just taking an average between the middle 3). I believe this would be a very fair way to do it (a Truncated mean as suggested by oNyx).

In either case, I think the scores should be finalized very shortly (I'm fine with whatever method the organizers decide to use), and then a plan set up for next year. I propose truncated average (remove lowest and highest score).

Excellent idea!
Offline appel

JGO Wizard


Medals: 68
Projects: 4


I always win!


« Reply #42 - Posted 2009-04-02 00:52:16 »

Although all these ideas are worth chatting about, I do not think applying them to the current results will help (the competition). Cooking more with the results will only do harm.

We will instead try to apply these ideas to next year's judging. Which reminds me, we should probably discuss the overall judging process instead of just how to calculate the percentages. This is probably a topic for another topic.

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline SimonH
« Reply #43 - Posted 2009-04-02 00:52:50 »

The score chart has now been corrected!
Noooo! Now 4x4k is 30th! This can't be right! 4x4K is much better than Jetp4k (now 26th)!
No offence to the judges but - WE NEED A BETTER METHOD!

EDIT: I'll tone this down later *hic* but s'true! *hic* *urp*

People make games and games make people
Offline appel

JGO Wizard


Medals: 68
Projects: 4


I always win!


« Reply #44 - Posted 2009-04-02 01:38:37 »

Noooo! Now 4x4k is 30th! This can't be right! 4x4K is much better than Jetp4k (now 26th)!
No offence to the judges but - WE NEED A BETTER METHOD!


IDEA ALERT

I've been thinking about the judging process and about this problem of "this game is better than that game" and using percentages to solve that. Doesn't seem like the best way.

IMO we should just have, let's say five bins, and allow the judges to put the games into in order to rate them, these could be:

Bin 1: SUPERB game
Bin 2: Excellent game
Bin 3: Good fun
Bin 4: Alright
Bin 5: Moving on

And within each bin, the judges could sort the games in the this-game-is-better-than-this-one order.

This could deliver much more meaningful results, as games are sorted better, and the daunting percentages are removed which actually hold no real meaning.

This also makes the judging process easier, as it's much more easier to put many games into just a handful of categories, and then deal with the games in each category.

Check out the 4K competition @ www.java4k.com
Check out GAMADU (my own site) @ http://gamadu.com/
Offline CyanPrime
« Reply #45 - Posted 2009-04-02 02:05:38 »

Virtual On 4K sucks T_______T
DeamonPants, you seem like you'd like Blue Fiend's new controls. You should re-check it out.
Offline Ranger
« Reply #46 - Posted 2009-04-02 02:09:52 »


IDEA ALERT

I've been thinking about the judging process and about this problem of "this game is better than that game" and using percentages to solve that. Doesn't seem like the best way.

IMO we should just have, let's say five bins, and allow the judges to put the games into in order to rate them, these could be:

Bin 1: SUPERB game
Bin 2: Excellent game
Bin 3: Good fun
Bin 4: Alright
Bin 5: Moving on

And within each bin, the judges could sort the games in the this-game-is-better-than-this-one order.

This could deliver much more meaningful results, as games are sorted better, and the daunting percentages are removed which actually hold no real meaning.

This also makes the judging process easier, as it's much more easier to put many games into just a handful of categories, and then deal with the games in each category.


I like it!  And it would be cool if users could also use this method to judge (but keep their score separate to the real judges scores).
Offline Morre

JGO Knight


Medals: 2
Projects: 10


I'm Dragonene on IRC.


« Reply #47 - Posted 2009-04-02 03:28:06 »

With the new method proposed, I feel the need to put n/a ("Couldn't run" or the like) is important. Apart from that... Yeah, I'm fine with the idea of having more of a bins sort of rating (1-5 or 1-10 per game).

I'm not sure there's much point sorting the entries within each bin if we're talking community vote - that should sort itself out with more than say, 10 or 15 votes. As for judges... that's trickier. Smiley

A method used rather successfully in the Ludumdare contest is to present each participant with a random games order judging page. People will generally start from the top, so if everybody rates say, a third of the games, each game will still get a decent amount of votes. There are problems with this method as well, but it's worked reasonably well for them I'd say. In order to be successful, however, it needs to be coupled with a strong encouragement for each participant to judge a few entries.

EDIT: Also, I feel like an idiot for having brought this whole 0-points issue up. Sorry about the mess! I've been coming out as all negative here, when all I really wanted was to point out the issue. I would've been entirely fine with the results table staying the same and new scoring mechanisms being used next year. With the new scoring table, two of my four games have an unfair advantage because darkfrog generally gave lower scores... :S This sort of problem is the reason we should remove lowest and highest scores for all games or normalize the judges' results.

I'm feeling really guilty right now.

Offline pjt33
« Reply #48 - Posted 2009-04-02 07:26:23 »

So then my comment wouldn't apply to your game then. Why quote it?
It was well after my bedtime and I wasn't expressing myself as clearly as I would like. My point was that, leaving aside the issue of which exception is thrown, we're targetting not one API/framework but several dozen, and we can't test on all of them. Lurking in the back of my mind also was the fact that in efforts to save a few bytes some people are straying into areas which the spec doesn't cover clearly. For example, AFAIK the spec for Applet.getGraphics() says nothing about it returning null at some stages in the applet's lifecycle, but some people found that that was the case with a small number of VMs.

Quote
Since the results have now been updated to ignore zero anyway, whats to worry about?
Next year, obviously.

Anyone else think percentage scores are unnecessarily precise given that the scores from each judge differed by over 50% in some cases. I think out of ten scores would probably be better for future contests.
darkfrog's standard deviation for presentation was 10.3, so with less precision nearly everyone would have been lumped together in three buckets under that scheme. In general most people scarcely use the bottom half of a 1-10 scale. I was thinking about something similar to appel's buckets suggestion, although that needs work to get something quantifiable which can be averaged.

Noooo! Now 4x4k is 30th! This can't be right! 4x4K is much better than Jetp4k (now 26th)!
In general there seems to be a moderate "nostalgia bonus" for basing it on a game the judges played in their youth.

EDIT: Also, I feel like an idiot for having brought this whole 0-points issue up. <snip>
I'm feeling really guilty right now.
Someone else would have brought it up. Relax.
Offline kevglass

« JGO Spiffy Duke »


Medals: 211
Projects: 24
Exp: 18 years


Coder, Trainee Pixel Artist, Game Reviewer


« Reply #49 - Posted 2009-04-02 07:45:09 »

Quote
It was well after my bedtime and I wasn't expressing myself as clearly as I would like.

Fair enough.

Quote
Next year, obviously.

Seems like that's already being thought about, so thats great too.

I won't be getting involved in the contest again. This sort of stuff afterwards just leaves what was fun activity with a nasty taste. The only good option is community voting and reviewing.

Kev

Offline cylab

JGO Ninja


Medals: 55



« Reply #50 - Posted 2009-04-02 07:59:03 »

I won't be getting involved in the contest again. This sort of stuff afterwards just leaves what was fun activity with a nasty taste.
Come on, that's late aprilsfool, isn't it. It's just natural to discuss the judging (and possible problematic scores) and I don't think this makes the excelent games this year any worse.

The only good option is community voting and reviewing.
Hmm, not sure about this. Should first be tried parallel to judge votes to see, if there are enough community votes/reviews for all games.

Mathias - I Know What [you] Did Last Summer!
Offline woogley
« Reply #51 - Posted 2009-04-02 08:10:38 »

This sort of stuff afterwards just leaves what was fun activity with a nasty taste.

Yeah, I agree. This quote always reminds me why I quit hosting:

lol, no good deed goes unpunished woogley Wink

Thanks appel! The torch is yours!!

Congrats to all game devs who wrote excellent games and kept the 4K fun. For those who bicker about pointless stats, please take that shit to the Flash 4K contest or something. It's not that big of a deal.
Offline Markus_Persson

JGO Wizard


Medals: 16
Projects: 19


Mojang Specifications


« Reply #52 - Posted 2009-04-02 08:12:29 »

I think having (from next year on) the judges simply list the games from best to.. uhm.. least best is actually a very good idea. It removes all subjective scoring from the equation and enforces a uniform point system. Having bins as well has the added benefit of distinguishing games into discrete groups, so there could be two AWESOME, a hundred VERY GOOD, ten OK, and two NOT OK.

I don't mind these discussion about scoring, but I'm very happy apple said there would be no more fiddling.

And don't quit, kev.. don't be like that.

[edit:]
It seems like mojang.com is down.. I can't find out why until I get home tonight. Did we get slashdotted or something?

[edit edit:]
Nevermind..

Play Minecraft!
Offline sunsett

Senior Devvie




ribbit!


« Reply #53 - Posted 2009-04-02 16:52:08 »

?! It probably didn't affect first place, but that's a much weaker statement than not affecting the outcome.

BTW darkfrog, could you say which version of Java you were using?

I don't remember which machine I was running yours on.  It was either Java 1.6u7 or Java 1.6u12, but I'm not sure. However, since 1.6u7 is standard on Mac and 1.6u12 is the latest version I think that though it sucks for you, it is fair to expect the game should run without issue.  Remember that we are judging a game that was coded and part of the coding is compatibility.  If you haven't tested your game to run properly on 1.6 or greater then you have to lose points.  It sucks that it drops you to zero, but like Chris said, what other score can I give if I can't play it?
Offline sunsett

Senior Devvie




ribbit!


« Reply #54 - Posted 2009-04-02 17:03:26 »

Wow!  Excellent Judging!  Very helpful reviews.  Big thanks to Appel, Chris, and Demonpants for the great comments!  Darkfrog, errr, thanks for playing it!  Smiley

Appel, the Java4K Result page looks great!  Excellent job!

Congrats to Markus for a well deserved victory.  Gloat away!

What game was yours? Was yours the one I didn't put any comment at all in? Shocked
Offline pjt33
« Reply #55 - Posted 2009-04-02 17:04:42 »

I don't remember which machine I was running yours on.  It was either Java 1.6u7 or Java 1.6u12, but I'm not sure. However, since 1.6u7 is standard on Mac and 1.6u12 is the latest version I think that though it sucks for you, it is fair to expect the game should run without issue.  Remember that we are judging a game that was coded and part of the coding is compatibility.  If you haven't tested your game to run properly on 1.6 or greater then you have to lose points.  It sucks that it drops you to zero, but like Chris said, what other score can I give if I can't play it?
It does run properly on the versions of 1.6 which were released when I finished it (i.e. up to 1.6u11). u12 came out in the last week of February, and u13 sometime in March, it seems, although I didn't know it existed until today. The breakage in u12 looks like a bug in Webstart, and u13 seems to have a different bug in Webstart based on the stack trace a friend sent me today.

P.S. Surely the latest version is 1.7?
Offline Orangy Tang

JGO Kernel


Medals: 56
Projects: 11


Monkey for a head


« Reply #56 - Posted 2009-04-02 17:06:21 »

I don't remember which machine I was running yours on.  It was either Java 1.6u7 or Java 1.6u12, but I'm not sure.

Interesting, was that what you used for NiGHTS and had problems with as well? Because I tested with 1.6 (and it's working on 1.6.0_07 32-bit WinXP here), so it sounds like there's something else going on. Huh

[ TriangularPixels.com - Play Growth Spurt, Rescue Squad and Snowman Village ] [ Rebirth - game resource library ]
Offline sunsett

Senior Devvie




ribbit!


« Reply #57 - Posted 2009-04-02 17:12:35 »

EDIT: Also, I feel like an idiot for having brought this whole 0-points issue up. Sorry about the mess! I've been coming out as all negative here, when all I really wanted was to point out the issue. I would've been entirely fine with the results table staying the same and new scoring mechanisms being used next year. With the new scoring table, two of my four games have an unfair advantage because darkfrog generally gave lower scores... :S This sort of problem is the reason we should remove lowest and highest scores for all games or normalize the judges' results.

I did in general give lower scores, but it would because I kept holding out for a game that stood out as exceptional to give high marks to.  Last year I marked everything relatively high and then near the end found a game that complete changed the standard and had to go back and move everything else down to differentiate it.  Please don't take that to mean the games were of bad quality, but rather for the most part most of them were pretty high quality, and though I didn't give 100s to anyone, the reason the score differentiation for me wasn't very great for most was because most of the games were of a generally high quality.  However, the fact that I scored everything relatively lower it shouldn't have any impact on the order of results, just the end scores.
Offline sunsett

Senior Devvie




ribbit!


« Reply #58 - Posted 2009-04-02 17:14:39 »

It does run properly on the versions of 1.6 which were released when I finished it (i.e. up to 1.6u11). u12 came out in the last week of February, and u13 sometime in March, it seems, although I didn't know it existed until today. The breakage in u12 looks like a bug in Webstart, and u13 seems to have a different bug in Webstart based on the stack trace a friend sent me today.

P.S. Surely the latest version is 1.7?

Latest stable version of Java is 1.6u12 and though it sucks to have this be an issue you have to be able to support the currently stable version of Java.  I would be curious to know what it is in your code that is causing a breakage in Java in u12 though, that's very odd.
Offline sunsett

Senior Devvie




ribbit!


« Reply #59 - Posted 2009-04-02 17:16:42 »

Interesting, was that what you used for NiGHTS and had problems with as well? Because I tested with 1.6 (and it's working on 1.6.0_07 32-bit WinXP here), so it sounds like there's something else going on. Huh

I did about half my testing on a Mac with 1.6u7 and the other half on Windows Vista 64-bit with 1.6u12.

Anyone that I gave a zero score for failing to run the application please feel free to PM me if you'd like me to help you resolve the issue.  I would also just like to be able to play your games. Smiley
Pages: 1 [2] 3
  ignore  |  Print  
 
 
You cannot reply to this message, because it is very, very old.

 

Add your game by posting it in the WIP section,
or publish it in Showcase.

The first screenshot will be displayed as a thumbnail.

rwatson462 (29 views)
2014-12-15 09:26:44

Mr.CodeIt (20 views)
2014-12-14 19:50:38

BurntPizza (40 views)
2014-12-09 22:41:13

BurntPizza (75 views)
2014-12-08 04:46:31

JscottyBieshaar (37 views)
2014-12-05 12:39:02

SHC (50 views)
2014-12-03 16:27:13

CopyableCougar4 (45 views)
2014-11-29 21:32:03

toopeicgaming1999 (113 views)
2014-11-26 15:22:04

toopeicgaming1999 (100 views)
2014-11-26 15:20:36

toopeicgaming1999 (30 views)
2014-11-26 15:20:08
Understanding relations between setOrigin, setScale and setPosition in libGdx
by mbabuskov
2014-10-09 22:35:00

Definite guide to supporting multiple device resolutions on Android (2014)
by mbabuskov
2014-10-02 22:36:02

List of Learning Resources
by Longor1996
2014-08-16 10:40:00

List of Learning Resources
by SilverTiger
2014-08-05 19:33:27

Resources for WIP games
by CogWheelz
2014-08-01 16:20:17

Resources for WIP games
by CogWheelz
2014-08-01 16:19:50

List of Learning Resources
by SilverTiger
2014-07-31 16:29:50

List of Learning Resources
by SilverTiger
2014-07-31 16:26:06
java-gaming.org is not responsible for the content posted by its members, including references to external websites, and other references that may or may not have a relation with our primarily gaming and game production oriented community. inquiries and complaints can be sent via email to the info‑account of the company managing the website of java‑gaming.org
Powered by MySQL Powered by PHP Powered by SMF 1.1.18 | SMF © 2013, Simple Machines | Managed by Enhanced Four Valid XHTML 1.0! Valid CSS!