who needs judges?

[moogie]

I am actually more keen on a few "experts" as judges as I am more interested in comments about different sections about the entries rather than who wins

I doubt anyone will put critical comments down if the voting is public.

[x30ice]

I like the different sub categories idea, but it may be hard to make a simple way to vote. Anyway, the best gameplay, best graphics, best technical achievement and best sound/music would be great. I think it would be better for this categories to get only one vote and not a top x.

If searching for the critical comments, I think too that public vote will stop lots of critical comments.

I think that if we make people vote, make people vote the games they played. I seriously doubt that all the people who may vote will play all the games, maybe played the ones they get attention for screenshots or name (for example) and then vote the best from the ones they played, so rather than a best 3/5/10 I think it would be better a list with all the games and then the voter rate the games that he/she played with the different categories required (pulldown menus to make it easy) and a final score that he/she thinks the game deserves (so the game may lack sound, for example, but it could score 10 on the final score) and finally a textbox for comments (maybe not required but very appreciated) . This way it could take a long time to vote for ones that played all the games but it would be more fair than a "top something" IMHO.

[Mojomonkey]

I think we should just text IDOL01 into our Cingular phones to vote for Latoya London!

After reading this thread, I believe the title is answered : "who needs judges?" The 4k Contest and it needs them badly.

Games published by our own members! Check 'em out!

[woogley]

agreed, sir monkey

volunteers?

[darkprophet]

If you decide to want judges this year again, i'll be more than happy to be one...

*cough* me! *cough*

DP

[blahblahblahh]

Hey, why not have judges?

Kev

Honestly, it's simple and works - it's transparent, in a way that voiting by anonysmous people isnt'. And you can be sure a handful of people almost certainly aren't cheating because people *know who they are*.

You *do* need to tell the judges in advance waht teh "objectvie" criteria are - this is the palce where it fell down last time (as noted before, TBH I think simply reading the judges comments from last time it would be easy to do it really well this time with minimal effort. Why re-invent the wheel, why not just improve? Shrug)

Finally...I'm speaking at the Casuality conference in a week's time. I would really really really like to be able to point people at 4k entries whenever I come across someone interested in it. Please, authors, make sure you've got something up by the 7th feb .

[blahblahblahh]

volunteers?

If you need an extra one, and we have a deadline for judging , and someone can write guidelines, and they actually all work this time , then I'm happy to do it again.

[jbanes]

You *do* need to tell the judges in advance waht teh "objectvie" criteria are - this is the palce where it fell down last time (as noted before, TBH I think simply reading the judges comments from last time it would be easy to do it really well this time with minimal effort. Why re-invent the wheel, why not just improve? Shrug)

While I would *like* to see public voting, I agree that we should have discussed this earlier rather than at the last minute. From where we're standing, judges are certainly the path of least resistence. Perhaps this year each judge should just give a 1-10 ranking instead of bothering with the individual components? Other than first place (T4XI), I don't think the results came out the way any of you three thought they should. At least with a simple ranking you can intentionally bin each game in the ranking you think it deserves.

What are your thoughts, Blah?

Finally...I'm speaking at the Casuality conference in a week's time. I would really really really like to be able to point people at 4k entries whenever I come across someone interested in it. Please, authors, make sure you've got something up by the 7th feb .

What do you need more entries for? Just point them at Xero and you're done! (I kid! I kid!) 

[woogley]

wow I have not seen blah^3 in a while! since you don't have a game in the contest adam, we'd love to have you as a judge. I'm trying also to get some of my gamer friends from Dell (that's where I work). those guys really don't care about the technology aspects, just if the game is good quality fun.

3 tech judges and 3 gamer judges would be a sweet balance

[Mojomonkey]

I'd be willing to judge depending on the time frame. I'm moving to Austin, TX mid-Feb, so will be indisposed for a few weeks.

Games published by our own members! Check 'em out!

[DonaldEKnuth]

3 tech judges and 3 gamer judges would be a sweet balance

Replace the 3:s with 10:s and it would be even sweeter! But I guess that's impossible.

[Morre]

Now, I would like to see public voting as well, and I still believe it can be done, and done well, if you do it the way I proposed, hehe. Anyway, giving the judges nothing but a 1-10 scale to work with would be the same as having public voting the way I proposed, but with less voters - I can't see how that improves the voting. No, you'd have to give them different things to rate, but surely the scales and criterion have to be improved compared to last year.

Judges are fine, but give them either a large scale or several scales - 1 to 10 for 30 games would force the judges to put several numbers on each scale, maybe leading to two games having 10 from all judges, thus making it a draw. That won't do

The more judges, the more unlikely this is. The bigger the scales, and the more criterion used, the more unlikely this is. Judges - sure, but make sure it's better coordinated than last year

[BitDragon]

Just jumped into this thread, perhaps somebody already proposed this:

Have the best of both worlds. Keep the judges, give them guidelines, try to get as objective a judgement as possible, for example, use principles from hci, like heuristic evaluation. In addition, offer a public vote, so people can rate their favorites and play votebot games if they like. With 50+ entries this year, I see no problem with having a "favorite of the audience" in addition to the official winners. This would also invalidate the argument that changing the evaluation method late in the contest is a bad move.

Cheers

Wolfgang

--
"I'm a f****** starship, I'm allowed to cheat!"
GCU Arbitrary, Culture Craft

[Morre]

Great idea!

[jojoh]

3 tech judges and 3 gamer judges would be a sweet balance

Replace the 3:s with 10:s and it would be even sweeter! But I guess that's impossible.

Yep!
I just had a look at last years votes and because there were so few judges, some games got heavily penalized because one judge killed it.
Neb 4K for example (one of my favs) got 9 from one judge and 21 and 14 from other. Quite a gap! Bubble Racer got 8 to 18 points, Trek 9 to 20 and Blast got 5 from Adam 14 from Scott and 20 from Chris  Average score from Scott and Chris is then 17 which would have given Blast a ... 2nd place! Instead it ended up with average 13.5 and place 19...

Judges can be good if there only are enough of them.

[woogley]

Judges can be good if there only are enough of them.

hence why I've been pushing for a (simple) public vote

[DonaldEKnuth]

Is it really necessary that the judges aren't competing? It could be a rule that they aren't allowed to rate their own game (which they shouldn't). If so I would gladly be one as well, I intend to try out every game anyway and I am a really fair guy!

[jbanes]

Is it really necessary that the judges aren't competing? It could be a rule that they aren't allowed to rate their own game (which they shouldn't).

It's a conflict of interest. Anyone entering the competiton is automatically going to want their entry to win. Whether they do it conciously or not, they may find themselves rating other games based a perception of how they rate in comparison to their own "Uber cool, super l33t!" title. Therefore, to maintain objectivity it has generally been required that the judge either:

A) Not have an entry
B) Remove his entry from judging

I believe the precident for B was set when Mlk offered to withdraw his entry so that he could judge. On the other hand, it was already known that his Stag Party 4K wasn't going to take top position. (He had some trouble with his 3D rotations, and really released it as one of those "Isn't this cool to play when you're drunk?" type of things. ) I'm not sure what the community would think about withdrawing a more likely contender.

[DonaldEKnuth]

I don't think that I am very likely to win, but I would like to know what the judges think. So I guess I can't join in and judge. There seem to be a shortage of judges though.

[arne]

If you can only vote by i.e. choosing the best three entries, you can't vote other entries down. Then, if you exlude your own entry from that list, I think it's only ok for competitors to judge, because they then give points to other entries -> so they more or less vote their own entries down.

[Markus_Persson]

I'd be willing to act as a judge if there's a shortage.

Yes, that means miners4k can't win, but I'm not in it for the competition. =)

[blahblahblahh]

Just jumped into this thread, perhaps somebody already proposed this:

Have the best of both worlds. Keep the judges, give them guidelines, try to get as objective a judgement as possible, for example, use principles from hci, like heuristic evaluation. In addition, offer a public vote, so people can rate their favorites and play votebot games if they like. With 50+ entries this year, I see no problem with having a "favorite of the audience" in addition to the official winners. This would also invalidate the argument that changing the evaluation method late in the contest is a bad move.

Yes to both. All the stuff about heuristics - pretty much what we said last time (I know I did, at least ). Public vote in parallel - simple first-past-the-post, see-how-it-goes. Why not?

[blahblahblahh]

wow I have not seen blah^3 in a while!

Yeah . Our game is too popular!

[woogley]

which game is yours?

[jbanes]

which game is yours?

I'm going to go out on a limb here, and say "Survivor". No, I didn't see Blah's sig. I have no idea what you're talking about. STOP LOOKING AT ME LIKE THAT! 

[kevglass]

Na, the game of the company he works for he means i think. Its an ARG and its outrageously popular.

Kev

[Anon666]

If you can only vote by i.e. choosing the best three entries, you can't vote other entries down. Then, if you exlude your own entry from that list, I think it's only ok for competitors to judge, because they then give points to other entries -> so they more or less vote their own entries down.

Like they do in the Eurovision?

That works sooooo well 

[DonaldEKnuth]

If there is a public vote it would be nice if the visitors could give grades on different aspects of the game (graphics etc) and also be able to provide a one sentence motivation for giving each grade. Those comments and grades would be really nice to have for the game authors.