I think the judges should certainly collaborate. When they worked alone, they came out with independent scoring systems that were incompatible with each other. Taking the averages of those numbers to determine rank never made any sense. In fact, the numbers themselves never really made any sense. Judges seemed to be able sort the list, but the game evaluation scores made little or no sense. The score categories made even less sense and none of the judges seemed to pay attention to them anyway. For example, there were many games that had very little game play value, but were technically very difficult to implement. I never saw that reflected in the judge's scoring. I never saw the judge write in his comments something to the effect of, well this game is no fun, but the 3D graphics are unbelievably amazing. Hence game-play = 10, technical = 95.
I think the most efficient way for the judges to collaborate would be to hold elimination rounds. The judges could separately create ranked lists. Then they could collaborate and agree together to discard half the entries. Then they could go back to the remainder and re-rank. This could be repeated until we have a top 5. The judges could argue together until they agree on ranking those top 5. The results of each phase could be posted and it would be interesting to see these partial results over time. In fact, a chat log between the judges arguing over which entries stay and which goes would be awesome to read.
I don't know if you're paying attention. From what I have seen of the system, doing this elimination would make very little difference. It's just a weighted average. If I give every single game a score between 90 and 100 (giving out nothing below 90), then effectively 90 is a 0 for me and 100 is still a 100 - I'm just reducing my effective range so I only have 10 options. Or, if I only give out a 0 through 10, then all 10s are basically 100s. You might see a 10 on my entry and a 100 on Appel's, but in terms of finding the winner nothing changes.
More importantly, I don't know if you actually
read all the reviews we painstakingly wrote (for free).
Here is a small sampling of reviews I did:
4bidden Fruit - Overall 96% Technical 86%This is a truly wonderful little 4k gem and I had a great time collecting all the crescents. One of the greatest strengths of 4bidden Fruit is its very calming atmosphere and simple gameplay. Despite the 4k file size, I really felt like I was playing in a fully fleshed out garden that hadn't been hampered by any such limitations. The crescents were just difficult enough to find that I was not bored or frustrated - instead I enjoyed climbing higher and higher in my attempts to find more. Some sort of radar would have aided the game, however, as towards the end it could get frustrating finding the last few crescents.
Technically the game is also quite impressive, mostly because of the multiple fractal generating algorithms you were able to pack into 4k. The control and gravity are fairly standard fare, however.
F-Zero 4K - Overall: 66% Technical 84%I think you did a great job capturing all the different elements of F-Zero into this little 4k game (except boost power), which is definitely an achievement. In addition, the dynamic 3D flat-down Mario Kart style level looks great, and the cars are very detailed for 4k.
I didn't feel that you really captured the sprit of F-Zero, nor was the gameplay particularly exciting. I think F-Zero should inspire a sense of extreme speed and craziness, but your game felt more like I should be driving a kart.
Frequent Flier - Overall: 68% Technical 91%I'm absolutely amazed you pulled it off in a 4k game. I (honestly) wouldn't even know how exactly to do this in any Java game.
Maybe I was singing out of its allotted range, but I couldn't help but to think that even a real singer would have trouble with this.
Ultimate Tic-Tac-Toe - Overall: 50% Presentation: 85%Well, there's not much to say about this. It's very slightly larger tic-tac-toe. The enemy AI is very strong (not bad for a solved game), to the point where I either tied or I lost. The balls reflecting each other looks really great... really, what else can I say about this?
4bsolution - Overall: 80% Presentation: 99%I found this game fairly entertaining but it tired me out relatively quickly. The idea is sound enough, and certainly has potential, but it all just takes too long and is too easy.
The presentation in 4bsolution is easily close to the absolute best this year. I really felt like a cool little critter swimming underwater gathering sentient little dots.
Are those examples perhaps indicative of the fact that "There were many games that had very little game play value, but were technically very difficult to implement. I never saw that reflected in the judge's scoring." is probably the result of you not actually reading all the reviews? From what I saw of every judge, if there was a game they thought overall wasn't very good but was very technically amazing, they gave out low overall and high technical. Also, when games looked great, they got high presentation. So.. what's your point, other than that you're going on about something that's not lacking?
One thing I will agree with you about is that a chat log or something like that would be very cool. If I were you I'd be very interesting in seeing the thoughts of my judges laid out on paper. Unfortunately, this is not feasible for the reasons pjt33 already mentioned. We're all over the world and we've all got limited time. Usually I play a few games when I'm on the train or something like that, every day to and from work. There is no internet when I'm on the train, so chats are not an option. Plus, last year there were so many games that Appel had to extend the deadline so many of us (including me) could finish up our reviews. Pushing it even further so that we could then discuss our favorites and all that seems like it would be in no way worth it. Had we discussed last year's entrants, we would have all overwhelmingly said that Left 4k Dead was the clear winner. And guess what won? We may have had a bit of argument about the next 4 places (my tie for first ended up in 4th place), but really the changes are so insignificant that it's not going to make a big difference. And, once again, having to get all the judges together at the same time would be borderline impossible.
PS - Last year's reviews are here, if you want to look over them again.
http://www.java4k.com/index.php?action=games&method=reviews&cid=5
