The Great Judging Debate 2014 Edition

Spoiler for:

Mistvale
   [7.5] Polish
   [5.5] Playability
   [6.0] Entertainment
   [5.0] Ingenuity

The polish and playability would have been much higher if the bugs (both code and visual) were not so prevalent. The reason that ingenuity is so low, is that while very lovely and extremely complex for a one week project ... the game was a fairly straight-forward RPG. Now, I realize that RPG Maker is tailored to making RPGs, but as the contest was focused on complete freedom, I had hoped entries would branch out from normal RPGs and do something fresh. I also would have liked if each character had their own story, instead of being a daughter or son of the mayor, but I can understand that time restraints would keep this from happening.

Okay, though there's no point of it: I'm going to make my case.

I didn't know we could lose points on polish for bugs. o.o Even the passability bugs are more based on gameplay though, right? But, fair enough. I can sorta maybe see the point on this one.

Also, all of the characters had to be the son/daughter of a mayor, since the main theme of the game was being the mayor and all that. :p Basically, the goal of the game is to customize your town and all that fancy stuff, but I guess the RPG aspects leaked out a bit more. I think Cornelius says something about it in his explanation of how to play the game, but I forget.

I'm trying not to be a butt about this, but giving us a 5/10 for ingenuity is a little insulting. Especially since most of our features weren't even explored - such as the Dating Sim aspects and the Town Customization. I didn't think that was a standard RPG, at least.

But bugs? Yeah, we had 'em. Lots and lots of bugs of many different species. They're always my downfall.

In past "Game in a week", I don't recalled it being said that you had to make an RPG. Some people didn't make their game an RPG. So why does freedom suddenly translate into "You either created a brand new genre, or your game sucks".

What does ingenuity even mean in this context? Telling a story without with little to no combat? Oh yeah, I've never seen that done before. I've never seen that done in RPG maker. How ingenious and ground breaking. 

So unless you can tell me how the two winning games "haven't been done before" then I'm calling BS on this entire thing.

o.o

o.o

Hmmmm... maybe i shall join yuyubabe next time...

I'm trying not to be a butt about this, but giving us a 5/10 for ingenuity is a little insulting. Especially since most of our features weren't even explored - such as the Dating Sim aspects and the Town Customization. I didn't think that was a standard RPG, at least.

My game also has bulding town/castle aspects, no fights, no history quests and a sim date system, of course the time ate us all and we could not show what it was alike, i feel good going 4th because it's the first time i do this crazy thing of making a game in a week.

I think i will change the graphics of the game and finish it and then release it... just for fun.

Thank you all!
Congrats Zylos!

And congrats to everyone!

My game also has bulding town/castle aspects, no fights, no history quests and a sim date system, of course the time ate us all and we could not show what it was alike, i feel good going 4th because it's the first time i do this crazy thing of making a game in a week.

Yeah, it's pretty tough stuff. Ironically, out of the three GIAWs I've competed in, the only game that won first place was the only game that I haven't taken seriously when making it. I think there's something to be said about too much dedication, perhaps.

I didn't know we could lose points on polish for bugs. o.o Even the passability bugs are more based on gameplay though, right? But, fair enough. I can sorta maybe see the point on this one.

I reasoned that audio and graphical issues went under Polish (preventing proper display), and scripting or eventing issues went under Gameplay (preventing the player to play through the game as intended).

Also, all of the characters had to be the son/daughter of a mayor, since the main theme of the game was being the mayor and all that. :p Basically, the goal of the game is to customize your town and all that fancy stuff, but I guess the RPG aspects leaked out a bit more. I think Cornelius says something about it in his explanation of how to play the game, but I forget.

Ah, honestly, I did not take away any points for any of that. I was just stating what I would have enjoyed seeing, is all.

I'm trying not to be a butt about this, but giving us a 5/10 for ingenuity is a little insulting. Especially since most of our features weren't even explored - such as the Dating Sim aspects and the Town Customization. I didn't think that was a standard RPG, at least.

I reasoned that each of those features settle into the Gameplay and/or Entertainment category. Also, I did play through the game more than once. I meandered through the other features on different playthroughs.

In past "Game in a week", I don't recalled it being said that you had to make an RPG. Some people didn't make their game an RPG. So why does freedom suddenly translate into "You either created a brand new genre, or your game sucks".

What does ingenuity even mean in this context? Telling a story without with little to no combat? Oh yeah, I've never seen that done before. I've never seen that done in RPG maker. How ingenious and ground breaking. 

So unless you can tell me how the two winning games "haven't been done before" then I'm calling BS on this entire thing.

I said I was looking for something fresh, not something no one has ever done before. Also, I did not automatically flunk any submitted game simply because the genre settled into the RPG category.


Now, looking back on everything, did I read too much into the freedom aspect of the contest, perhaps turning the open idea into an actual theme? I suppose I did. Under the Ingenuity section I read : "originality and story: your usage of FREEDOM" and I saw that as a banner for aiming toward something fresh. I admit, I made an error in judgement.

Perhaps I was far too harsh. Perhaps I made mistakes in my reasoning. Perhaps I should have chosen better wording than 'standard RPG', which is not intended as a slight.

I am just a volunteer judge. And, in fact, this is my first time judging a gaming contest. Or any contest. Ever. Not to worry though, I will not be doing so in the future.

< GIAW XII Exhydra's Scores with Brief Notes >

Spoiler for:

Lost of Victorious End
   [2.5] Polish
   [1.5] Playability
   [1.0] Entertainment
   [1.0] Ingenuity

Alright. I have some helpful advice for the author of this game : start with the very basics first.

The game should be much more than a collection of nifty scripts and the functionality that they add. Focus your effort on dialog and story elements. If English is not your native language, that is all the more reason to focus on what your characters are saying. Next, move on to placing and polishing graphics. And if you absolutely need a script, then add it. Do not simply add a script you think you might use later. Add the script only when you do need it.

Also, including LGBT aspects is fine. Centering the story around the sexuality of one or more characters is fine. But when everyone is either flinging around degrading words or praising <insert sexuality>, I start flipping desks. Include more middle ground. Use subtle wording or actions. Anything but blatant obscenities, unless you absolutely need to.

I think that somewhere in your mind, there is a story or stories to be told. But, you seem intent on trying to impress others with graphics and fancy scripting. While proper usage of the aforementioned can impress people, I am telling you now that a good story will earn you more in the end. Practice, practice, practice!

Well, there should be added more Story so far, but after four days I quitted allready, because I didn't had any Internet and got depressed of missing the upload date. I'm in really bad conditions these days and to make all graphics and bug testing exhausted me. After the date there should be an attack and the real story should happen. But I was way to down, sorry for that. :'(

Normally the scripts should be all in use in the demo, the rest read above.

I'm sorry, if I offended the judges with it, didn't meant to. I only kinda tried to implement my real life situation in the game, sorry. :'( (\s/)

Well, after all I didn't expected much, but I wanted at least to send anything in... /)

But may I ask, why I got such a bad playability?
I thought I fixed all bugs and stuff. :/

In past "Game in a week", I don't recalled it being said that you had to make an RPG. Some people didn't make their game an RPG. So why does freedom suddenly translate into "You either created a brand new genre, or your game sucks".

What does ingenuity even mean in this context? Telling a story without with little to no combat? Oh yeah, I've never seen that done before. I've never seen that done in RPG maker. How ingenious and ground breaking. 

So unless you can tell me how the two winning games "haven't been done before" then I'm calling BS on this entire thing.

I am just a volunteer judge. And, in fact, this is my first time judging a gaming contest. Or any contest. Ever. Not to worry though, I will not be doing so in the future.

My rant wasn't aimed at you, and I apologize if it seemed that it was. It was just me letting off some steam over the score. I just want you know that it's not really anything personal, but that I just don't agree the score.  I don't think you're a bad person and everyone has differing opinions and will score/interpret things differently.

Spoiler for:

I was debating if I should use or and I decided on but now I'm thinking I should have use instead.

EDIT: plus, I want your body, Exhydra

My rant wasn't aimed at you, and I apologize if it seemed that it was. It was just me letting off some steam over the score. I just want you know that it's not really anything personal, but that I just don't agree the score.  I don't think you're a bad person and everyone has differing opinions and will score/interpret things differently.

Ah, well, just to be clear, as judges we did not discuss or debate criteria, nor did we share any scores amongst ourselves. At least, no discussion that I was privy to. So, what I posted above is simply my interpretation, and might not be shared by any other judge.

So unless you can tell me how the two winning games "haven't been done before" then I'm calling BS on this entire thing.

My main mechanics were teleportation, force pushing, force pulling and the concept of playing as a beta tester testing a game by multiple different authors. Where has that been done?

the Beta Tester
   [6.0] Polish
   [6.5] Playability
   [6.5] Entertainment
   [7.0] Ingenuity

 I totally get the first three. I had no time to test for balance or bugs and I didn't have an artist. Totally get it. Completely fair scores. But what would have been a 10 for Ingenuity? To me a 7.0 is a C-...I didn't even use RPG maker battles at all. I don't mean to hound on your scores, I know judging can be rough, but...C-?

As an outsider looking in. This would make me really leery of ever being a judge in a GIAW.  Perhaps the guidelines for what you expect in judging should be a little more clear?  Just a thought 

I always kinda thought that polish was pure looks and sounds, and everything bug-related was gameplay. o.o But, it's possible that I could be entirely wrong. Maybe there isn't a right or wrong way to score, though. The guidelines are a little blurry and perhaps we need to be more specific with them in the future. (Can't get much worse than the "genre" guideline last time. THAT's what I call blurry). That way no one can mistake them. That goes for me, too. B( I'm sure I've made many mistakes with them.

I guess I was mostly shocked with the ingenuity thing. I didn't think the story was 100% original, because I was going for something simple this time, and certainly not as story-based as Major Arcana. But, I didn't think it was that bad. And I was kind of proud of the way it was executed. But it's really not a big deal. I can see your points, and I know everyone is going to have a different opinion.

EDIT: plus, I want your body, Exhydra

o.o

Ah, well, just to be clear, as judges we did not discuss or debate criteria, nor did we share any scores amongst ourselves. At least, no discussion that I was privy to. So, what I posted above is simply my interpretation, and might not be shared by any other judge.

Also...wait, what? In the last GIAW, the judges were always discussing things. o.o Maybe we were just the odd ones, though. o.o We still made a couple of mistakes, with that stupid "genre" scoring guideline. ;_;


All of that aside: I don't think any of the comments above should impact your perception of how you are as a judge. To be honest, I think we get something like this almost every time a judge posts their specific results. And we will almost every time a judge does post them. There's always going to be some jerk (like me) that will argue with the interpretations. Don't let that deter you, okay?

(Yes, I know I'm being hypocritical and awfully wishy-washy. ;_; )

Here's how I handled my scores.

Polish is an overall measure of how completed, how professional the game is.  If I find bugs, it makes the game look less polished, ditto with bad maps, or low quality music, badly cut sprites, poor dialogue, poor game balance, et cetera.  The more fragile your game appears, the less polish points I give it.

Playability is an overall measure of how the game flows.  If a game cannot be continued due to game breaking bugs, poor balance, lack of direction, or poor direction, it will lose playability points.  The smoother your game plays, the less the progression hiccups, the more playability points I give it.

Entertainment is self explanatory.  If I had fun, you score points, if I hated it, then low points.

Ingenuity is the one that seems to be tripping people up, so I'll give my take.  Ingenuity is an overall measure of how fresh the game felt, and how clever it is.  It doesn't matter if the core concepts are tried and true if they are applied in a fresh and interesting way.  See, all games, all stories have basically been done.  We can't grade this based on "I've never seen it before" or "I've seen it before."  Clever use of eventing systems will net you ingenuity points.  Clever applications of gameplay as a storytelling method will net you ingenuity points.  Clever storylines will net you ingenuity points.  Clever use of artwork and music will net you ingenuity points.  This category represents breaking away from the RPG Maker RTP standard and doing something that takes effort, thought, and...well, ingenuity.  It is NOT plain and simply originality but originality does factor in.

---

The entire purpose behind this four-category judgement system was to reward people who put in a lot of effort even if the judge didn't necessarily have fun.  In early GIAW events, entertainment bias riddled the scores, and otherwise well made games were given terrible scores because some of the judges didn't enjoy them.  This system also allows for flexibility, such as with "the beta tester" where for example, polish cannot be judged as black and white, due to the concept.

I hope this clears up my perspective at least.

Congratulations everyone! Everyone who finished has achieved a pretty remarkable thing as far as I'm concerned, so don't be dissuaded by low scores.

@Exhydra - Your scoring sounded fair enough to me and was aligned with the rubric given by Pac. The most important thing is to apply the same rubric to all the games, and as long as you do that it should even out when it is averaged with the other judges. Don't let it deter you from participating again. It compares quite favourably to some of my early judging; I was a pretty big jerk sometimes (Sorry corbin-nunn :'().

@yuyubabe - I understand your concerns, but any scoring system is going to have some subjectivity, and that comes through a bit in the ingenuity category. I'm sure that other judges found your game very ingenious, and perhaps games those judges considered boring ripoffs would have impressed Exhydra. Part of the reason there are so many judges is to ensure that one judge's idiosyncracies don't unfairly skew the score. Anyway, keep in mind that all of Exhydra's ingenuity scores were fairly low (and most lower than what he gave you), so it didn't hurt your average relative to the other competitors.

Also, I haven't played the game so I can't comment on this specifically and I might be misinterpreting your comment about some features being unexplored, but it's OK for a judge not to give points for planned features that aren't yet implemented or are only signalled toward. They have to judge what you made in a week, not what you could have made if you had a month.

Also, bugs can affect Polish or playability in my opinion, depending on whether the bug is audio/visual or gameplay-based. For instance, if your butterfly is turning into an ogre every time the player interacts with it from the wrong direction, then playability isn't affected at all, but it still disrupts the experience and is a sign of a lack of polish. (Well, at least when it's unintentional. If its intentional and makes sense, it might net you points for ingenuity with some judges. ).

@Pacman - I recommend that you post the averages for each judging category, and not just the total average. It might allay some concerns.

I totally get the first three. I had no time to test for balance or bugs and I didn't have an artist. Totally get it. Completely fair scores. But what would have been a 10 for Ingenuity? To me a 7.0 is a C-...I didn't even use RPG maker battles at all. I don't mean to hound on your scores, I know judging can be rough, but...C-?

I guess I was mostly shocked with the ingenuity thing. I didn't think the story was 100% original, because I was going for something simple this time, and certainly not as story-based as Major Arcana. But, I didn't think it was that bad. And I was kind of proud of the way it was executed. But it's really not a big deal. I can see your points, and I know everyone is going to have a different opinion.

I basically followed how tSwitch laid out Ingenuity ...

Ingenuity is the one that seems to be tripping people up, so I'll give my take.  Ingenuity is an overall measure of how fresh the game felt, and how clever it is.  It doesn't matter if the core concepts are tried and true if they are applied in a fresh and interesting way.  See, all games, all stories have basically been done.  We can't grade this based on "I've never seen it before" or "I've seen it before."  Clever use of eventing systems will net you ingenuity points.  Clever applications of gameplay as a storytelling method will net you ingenuity points.  Clever storylines will net you ingenuity points.  Clever use of artwork and music will net you ingenuity points.  This category represents breaking away from the RPG Maker RTP standard and doing something that takes effort, thought, and...well, ingenuity.  It is NOT plain and simply originality but originality does factor in.

... except, I believe I graded that category in particular rather harshly. In hindsight, I probably should have doled out more eights and even a nine, or so.

Also, I probably should have discussed my thoughts on the scoring criteria with the other judges. Ah well, live and learn ...

I will post my scores and thoughts.

But I will soon after change my name and go into hiding.

But may I ask, why I got such a bad playability?
I thought I fixed all bugs and stuff. :/

I don't know why I allways get ignored everywhere, may I'm really a push over. :'(
That was the only question I wanted to be answered. :/ (\s/)

Those are some pretty solid explanations. o.o I still think it might be good to keep a specific rubric somewhere, just for those that get confused by them. Not just to anyone specific, but I would like to use to use that for the next time I judge, too.

Also, I haven't played the game so I can't comment on this specifically and I might be misinterpreting your comment about some features being unexplored, but it's OK for a judge not to give points for planned features that aren't yet implemented or are only signalled toward. They have to judge what you made in a week, not what you could have made if you had a month.

The features were ones that were basically fully implemented, but were not shown in the video review. :p So, after watching the video, I thought that he didn't test out the features initially. Then, he informed us that he played the game more than once and did test out the features. Just a bit of a misunderstanding there.


Bluntsword: ;_; You don't have to post what you gave everyone specifically. In fact, it's often a bad idea because then people will do what I just did and try to argue the scores.

I just want people to like everyone.

o.o

I don't think anyone could ever not like you, bluntsword. ;_;

You're so likable.

;_______;

But may I ask, why I got such a bad playability?
I thought I fixed all bugs and stuff. :/

Ah, my apologies, your question got buried.

There were still bugs present. The woman in the top right of the cafe caused an infinite, inescapable Parallel Process conversation. The one-step average per random battle in the Training Area was ... not game ending, but certainly interrupted the flow of the game. I had to disable the random battle in the area to go in and speak to the NPCs in a timely fashion. Also, on the boardwalk where Nagisa or Asuka is waiting, one of the character graphic files was not selected properly, causing the game to crash. There were also a host of minor, non-game breaking issues, as well.

But may I ask, why I got such a bad playability?
I thought I fixed all bugs and stuff. :/

I don't know why I allways get ignored everywhere, may I'm really a push over. :'(
That was the only question I wanted to be answered. :/ (\s/)

I can't speak for the other judges, but it was confused me more than it was buggy.

But may I ask, why I got such a bad playability?
I thought I fixed all bugs and stuff. :/

I don't know why I allways get ignored everywhere, may I'm really a push over. :'(
That was the only question I wanted to be answered. :/ (\s/)

I can't speak for the other judges, but it was confused me more than it was buggy.

Yes, that too.

I'm sorry, if I offended the judges with it, didn't meant to. I only kinda tried to implement my real life situation in the game, sorry. :'( (\s/)

Also, I was not personally offended at all. Using a video game to vent can be rather therapeutic. And for inspiration, using real life situations can work well. However, unless you temper and tame the anger, everything will come out in a harsh jumble. No worries, just keep at it!

Maybe solidify the criteria a bit?:

Balance - Was the difficulty curve fair to a player just starting the game? Did it present challenges that made no sense in the context given? This is not "omg this game is too hard", it's "Is this game providing me with the knowledge necessary to beat the challenges it presents?" Some games are hard. Some people don't like hard games. This doesn't make the game worse.

Polish - Was the map design strong? Was the music fitting? Where there sound effects in the right places? This is not "Omg they used all default rpg maker stuff burn him" it's "Does it work? Does it set a mood properly? Does it feel right? Does this piece of music make sense for this situation? Do the sound effects make the scene more believable?"

Bugginess - Were there textures you could walk through? Did you get any error messages? Did the game stall at any points?

Mechanics - Are the mechanics being used in interesting ways? Are they repeated over and over again? This is not "Have I seen this mechanic before". For example, if the mechanic is smacking things, do you always smack the same things? Or do you start smacking new things that make it interesting?

Conveyance - Do you have a decent idea of where to go at all times? Is it clear what everything does? Was there ever a point where you said "wtf do I do?" or "why is that happening...?" When you fail/succeed, do you understand why and have a decent idea of what you could have done better?

Notice that there is no "fun" category. Fun is completely 100% subjective. So is difficulty and originality. These above criteria are things that are much more objective and could be decided by a few judges rather than a bunch, which inevitably causes judging to take a long time. Also, none of the criteria overlap, so failing at once aspect doesn't deplete points from multiple areas. For example, having a buggy game in this competition takes away points from two or more categories.