RMRK is retiring.
Registration is disabled. The site will remain online, but eventually become a read-only archive. More information.

RMRK.net has nothing to do with Blockchains, Cryptocurrency or NFTs. We have been around since the early 2000s, but there is a new group using the RMRK name that deals with those things. We have nothing to do with them.
NFTs are a scam, and if somebody is trying to persuade you to buy or invest in crypto/blockchain/NFT content, please turn them down and save your money. See this video for more information.
The Great Judging Debate 2014 Edition

0 Members and 2 Guests are viewing this topic.

*
The Hero of Rhyme
Rep:
Level 83
( ͡° ͜ʖ ͡°)
Project of the Year 20142014 Best RPG Maker User - Story2014 Queen of RMRK2011 Best Newbie2014 Best RPG Maker User - Creativity2014 Kindest Member2013 Queen of RMRKBronze SS AuthorBronze Writing ReviewerSecret Santa 2013 ParticipantFor taking arms in the name of your breakfast.GOOD!For frequently finding and reporting spam and spam bots2012 Best Yuyubabe Smiley2012 Best RPG Maker User (Creativity);o
I actually really like those. A lot. It's much more objective, a bit less opinion-based, and a bit more fair. Also, it creates a category that I think is a bit overdue - "where the hell do I go next"?

The one thing it needs is at least something to do with story/writing/dialogue (or an alternative for when a game has none of these). ;o Otherwise, the writers will be at a disadvantage!

(Note that "story" is originally supposed to be a part of ingenuity - not just originality!)

But other than that, I think we're onto something here.
Spoiler for My Games and Art:
ℒℴѵℯ❤


My Artwork Thread

The Lhuvia Tales [Current]

Ambassador [Complete]

The Postman [Complete]

The Wyvern [Complete]

Phoenix Wright: Haunted Turnabout [Complete]

Major Arcana [Cancelled]


********
Furry Philosopher
Rep:
Level 94
Rawr?
2013 Best RPG Maker User (Creativity)Randomizer - GIAW 11Gold - GIAW 11 (Hard)Secret Santa 2013 ParticipantFor frequently finding and reporting spam and spam bots2012 Best RPG Maker User (Programming)2012 Best RPG Maker User (Mapping)Secret Santa 2012 ParticipantGold - GIAW 9Project of the Month winner for September 2008For taking a crack at the RMRK Wiki2011 Best RPG Maker User (Programming)2011 Best Veteran2011 Kindest Member2010 Best RPG Maker User (Events)2010 Best RPG Maker User (Story)
Personally, I've always felt the categories were fine as is since they gave enough of the vital points while still leaving room for interpretation, but it has been a while since they were introduced so an update may be necessary sometime soon. The behind-the-scenes people who organize the contests will have to argue it out before the next contest.




********
Hungry
Rep:
Level 96
Mawbeast
2013 Best ArtistParticipant - GIAW 11Secret Santa 2013 ParticipantFor the great victory in the Breakfast War.2012 Best Game Creator (Non-RM Programs)~Bronze - GIAW 9Project of the Month winner for August 2008Project of the Month winner for December 20092011 Best Game Creator (Non RM)Gold - GIAW Halloween
Maybe solidify the criteria a bit?:

Spoiler for:
Balance - Was the difficulty curve fair to a player just starting the game? Did it present challenges that made no sense in the context given? This is not "omg this game is too hard", it's "Is this game providing me with the knowledge necessary to beat the challenges it presents?" Some games are hard. Some people don't like hard games. This doesn't make the game worse.

Polish - Was the map design strong? Was the music fitting? Where there sound effects in the right places? This is not "Omg they used all default rpg maker stuff burn him" it's "Does it work? Does it set a mood properly? Does it feel right? Does this piece of music make sense for this situation? Do the sound effects make the scene more believable?"

Bugginess - Were there textures you could walk through? Did you get any error messages? Did the game stall at any points?

Mechanics - Are the mechanics being used in interesting ways? Are they repeated over and over again? This is not "Have I seen this mechanic before". For example, if the mechanic is smacking things, do you always smack the same things? Or do you start smacking new things that make it interesting?

Conveyance - Do you have a decent idea of where to go at all times? Is it clear what everything does? Was there ever a point where you said "wtf do I do?" or "why is that happening...?" When you fail/succeed, do you understand why and have a decent idea of what you could have done better?

These categories appear to be really rigid to me...and they focus almost entirely on gameplay oriented games.  We need a judgement criteria that is flexible enough to accommodate ANY sort of game.

Notice that there is no "fun" category. Fun is completely 100% subjective. So is difficulty and originality. These above criteria are things that are much more objective and could be decided by a few judges rather than a bunch, which inevitably causes judging to take a long time. Also, none of the criteria overlap, so failing at once aspect doesn't deplete points from multiple areas. For example, having a buggy game in this competition takes away points from two or more categories.

Entertainment value needs to be part of the judgement criteria.  Whether it's a game for fun, or a serious game, or whatever...if you're hooked, then the game is GOOD and that needs to be part of the score, regardless of if it's a buggy mess.  Perhaps we should rename this as 'Immersion' to make it appear more solid, and allow for more broad application of the intended meaning of the category.

Playability could use a little tweaking though, possibly.

I actually really like those. A lot. It's much more objective, a bit less opinion-based, and a bit more fair. Also, it creates a category that I think is a bit overdue - "where the hell do I go next"?

That's covered in playability.  If you don't know what you're supposed to do, then the game isn't playable.  I think we do need an opinion category for judges to voice their feelings, as when we didn't, people did it anyway.

The one thing it needs is at least something to do with story/writing/dialogue (or an alternative for when a game has none of these). ;o Otherwise, the writers will be at a disadvantage!

(Note that "story" is originally supposed to be a part of ingenuity - not just originality!)

I don't think we need to make a flip-flop category for games with or without story, that is just overcomplicating the system.  As it stands, a well written story could make the game more polished, more entertaining, and more ingenious.  Additionally, good dialogue, I feel, falls under polish, and good direction from dialogue falls under playability.

I seriously don't think we should make a 'story' category, or adopt a strict standard of judgement, because that just further pigeonholes people into making story based RPGs that are gameplay focused.  If we say "I want to see these things in a game" and write the system based on that, we penalize anything else.

That's not fair.

FCF3a A+ C- D H- M P+ R T W- Z- Sf RLCT a cmn+++ d++ e++ f h+++ iw+++ j+ p sf+
Follow my project: MBlok | Find me on: Bandcamp | Twitter | Patreon

****
Rep:
Level 43
Somewhat got a project? (\ô/)
GIAW 14: ParticipantParticipant - GIAW 11
Ah, my apologies, your question got buried.

There were still bugs present. The woman in the top right of the cafe caused an infinite, inescapable Parallel Process conversation. The one-step average per random battle in the Training Area was ... not game ending, but certainly interrupted the flow of the game. I had to disable the random battle in the area to go in and speak to the NPCs in a timely fashion. Also, on the boardwalk where Nagisa or Asuka is waiting, one of the character graphic files was not selected properly, causing the game to crash. There were also a host of minor, non-game breaking issues, as well.
Damn, there you right, I overseen these parallel process and forgot to deactivate the random battle again in the Trainings Hall. I made it for balancing testing, but forgot to turn it back, my bad. :/
Well, normaly you should date her there and I wanted it to add on day 5, but I quitted early, because I didn't had internet and was not sure if I will get it in time. :'(

But now I can understand and agree 100% with your rating. There is nothing more annoying then gamebreaking bugs. :'(

Spoiler for:
For the Rating overall it would be great if it would be detailed like in game reviews, kinda like:
Graphic = How good or fitting is the optical aspect, or is it clashing?
Sound = Are the sounds and music that is used fitting to the situation/locations?
Difficulty = Is it way too easy or way to hard, or is it just right?
Story = Is the story clear to understand, interresting, or is it boring and does it even have a story.
Ingenuity = Is there anything new in it, or is it used in a new way?
Bug Density = Is it a super buggy game with gamebreaking stuff, or are only a few, or not even a bug at all?

And from these points you will get to the total rating. I think a 100% Rating would be more clear then a 0/10 Point system. It also will give more room for the judges to give a rating.  So if some people are close together from a rating one or two percents could make the difference then. And I think the chances would be for all the same, because one will get more in that point and others will get more in other points. ;) (\s/)

********
Hungry
Rep:
Level 96
Mawbeast
2013 Best ArtistParticipant - GIAW 11Secret Santa 2013 ParticipantFor the great victory in the Breakfast War.2012 Best Game Creator (Non-RM Programs)~Bronze - GIAW 9Project of the Month winner for August 2008Project of the Month winner for December 20092011 Best Game Creator (Non RM)Gold - GIAW Halloween
I just seriously can't agree with a judging criteria that assumes that a game needs to have some aspects.  I've played amazing games without sounds, without graphics, without stories, without gameplay, not all at once but you see what I'm getting at here?

FCF3a A+ C- D H- M P+ R T W- Z- Sf RLCT a cmn+++ d++ e++ f h+++ iw+++ j+ p sf+
Follow my project: MBlok | Find me on: Bandcamp | Twitter | Patreon

*
my name is Timothy what's yours
Rep:
Level 79
Hello
2014 Most Missed Member2014 Zero to Hero2014 Best IRC Quote2012 Zero To HeroSecret Santa 2012 ParticipantContestant - GIAW 9For frequently finding and reporting spam and spam bots2011 Zero to Hero
And from these points you will get to the total rating. I think a 100% Rating would be more clear then a 0/10 Point system. It also will give more room for the judges to give a rating.  So if some people are close together from a rating one or two percents could make the difference then. And I think the chances would be for all the same, because one will get more in that point and others will get more in other points. ;) (\s/)

Some judges already give scores with up to two digits after the decimal, and I rounded them to the nearest tenth (effectively being out of 100).

As for your proposed scoring categories, I already decided it was wrong when I read "story". I feel as though you just unpacked the already existing categories and threw in some elements which are way too subjective. Desired difficulty is different for every person, and isn't even a measure of how good a game is in the first place. There's no reason difficulty should be its own category when an extreme difficulty could result in a lower playabality score, perhaps, and a game with no difficulty could lose points in entertainment, for example.
Story can never be a category on its own for judging purposes. Some games don't have stories. On purpose. And they're perfectly fine games. In fact, one of the five games I consider to be perfect has exactly no story, and all of the others convey story through gameplay and little through dialogue. It's silly to score a game on how "good" its story is, when there's so much more to story in games than just the plot (and also it's all incredibly subjective).
The way you've described graphics and sound make it seem as though an Atmosphere category could cover those elements and more. However, again, treatment to atmosphere is subjective, so we bundle graphics and sound in to a more concise and objective category, Polish.
Bug Density is a factor of Polish and Playability. There are more things that can break a game than bugs, though, so we have to take them in to consideration as well, which is why it's not its own category.

All this is fine for reviewing games, but not judging them in a contest. It seems a bit basic, arbitrary and dangerously subjective. We try to bundle distinct categories that can include whatever aspects the game does or does not have. This is why we have broader terms than one might expect - to be all-encompassing but also concise. It's tricky, but you just need to forget what you think is important in a game and think about the experience itself.
it's like a metaphor or something i don't know

*
Rep:
Level 87
I just seriously can't agree with a judging criteria that assumes that a game needs to have some aspects.  I've played amazing games without sounds, without graphics, without stories, without gameplay, not all at once but you see what I'm getting at here?

But that's the beauty of it. It doesn't say "Is the music super awesome and have guitars" it says "Does it feel right". Sometimes, silence is exactly what builds up to something perfectly. Sometimes having no graphics at all to play as a blind man is more engaging than having graphics. Games without stories, such as Super Smash Bros, work great. Maybe you play as a deaf guy and can't hear the sounds at all. There's a few ways I could think that would be interesting. There are a ton of times when I wish the designers just shut off the music for a certain scene. That is part of polish. Knowing when to use something and when not to.

Quote
These categories appear to be really rigid to me...and they focus almost entirely on gameplay oriented games.  We need a judgement criteria that is flexible enough to accommodate ANY sort of game.


Gameplay is at the core of games. You also have to remember that as long as the player has control, there is gameplay. Choosing a story path is gameplay. As we've seen with a lot of story based games like Heavy Rain and Mass Effect, that can be done amazingly. Heavy Rain even does some amazing work with quick time events and Balance can easily be applied to that game's core mechanics. Even if the game has nothing but dialogue choices, the category is about making informed decisions. Do the dialogue options give you the information necessary to make the right choices or more broadly, the choices you want? If your game is 100% out of the player's control, you've created a movie.

Quote

Entertainment value needs to be part of the judgement criteria.  Whether it's a game for fun, or a serious game, or whatever...if you're hooked, then the game is GOOD and that needs to be part of the score, regardless of if it's a buggy mess.  Perhaps we should rename this as 'Immersion' to make it appear more solid, and allow for more broad application of the intended meaning of the category.

Playability could use a little tweaking though, possibly.


I tested my game with a few people after the contest. I had a few that loved it and a few that hated it. One person said "I'm just not into this sort of game. I don't like action games." Should my score be worse because one of the judges doesn't like this type of game? I hate RTSs, but should Starcraft II be taken off the shelves?

Quote
That's covered in playability.  If you don't know what you're supposed to do, then the game isn't playable.  I think we do need an opinion category for judges to voice their feelings, as when we didn't, people did it anyway.

That's not very clear though. Playability covers a LOT of aspects in the game currently. It covers "Gameplay and General Mechanics". What about them? I'm just suggesting focusing the criteria down so the game makers know what they will be judged on and the judges know what they are judging.


Quote
I don't think we need to make a flip-flop category for games with or without story, that is just overcomplicating the system.  As it stands, a well written story could make the game more polished, more entertaining, and more ingenious.  Additionally, good dialogue, I feel, falls under polish, and good direction from dialogue falls under playability.

I seriously don't think we should make a 'story' category, or adopt a strict standard of judgement, because that just further pigeonholes people into making story based RPGs that are gameplay focused.  If we say "I want to see these things in a game" and write the system based on that, we penalize anything else.

That's not fair.

I think that's a stretch to say that a good story falls under two categories. It also makes it so having a bad story hurts you BIG time. Having one bad thing in your game shouldn't punish you harder than having something else bad in your game.

The one thing it needs is at least something to do with story/writing/dialogue (or an alternative for when a game has none of these). ;o Otherwise, the writers will be at a disadvantage!

(Note that "story" is originally supposed to be a part of ingenuity - not just originality!)

What about this?:


Balance - Was the difficulty curve fair to a player just starting the game? Did it present challenges that made no sense in the context given? If you died, were you laughing and smiling or frustrated? This is not "omg this game is too hard", it's "Is this game providing me with the knowledge necessary to beat the challenges it presents?" Some games are hard. Some people don't like hard games. This doesn't make the game worse.

Polish - Was the map design strong? Was the music fitting? Where there sound effects in the right places? This is not "Omg they used all default rpg maker stuff burn him" it's "Does it work? Does it set a mood properly? Does it feel right? Does this piece of music make sense for this situation? Do the sound effects make the scene more believable?"

Bugginess - Were there textures you could walk through? Did you get any error messages? Did the game stall at any points?

Mechanics - Are the mechanics being used in interesting ways? Are they repeated over and over again? This is not "Have I seen this mechanic before". For example, if the mechanic is smacking things, do you always smack the same things? Or do you start smacking new things that make it interesting?

Conveyance - Do you have a decent idea of where to go at all times? Is it clear what everything does? Was there ever a point where you said "wtf do I do?" or "why is that happening...?" When you fail/succeed, do you understand why and have a decent idea of what you could have done better?

Build Up - Is there a feeling that the game escalates? Is the ending satisfying? Were the way encounters (of any kind) introduced powerful in their tone? This is not, "omg explosions this game is epic" this is, if the game has a dark mood, did the build up to the Shadow Fiend at the end of the game feel right. If the game is a dating sim and you are trying to go out with the super hunk from your school, did the build up to the date feel right. Overall, this is, "did the game make you feel engaged or excited at some point during the game?" If a boss just comes out of no where, that's not build up. If a boss is introduced before hand, teases you a bit and finally kills your best friend at the top of the mountain and you have to solo two of them, now we have build up.


*
Rep:
Level 87
And from these points you will get to the total rating. I think a 100% Rating would be more clear then a 0/10 Point system. It also will give more room for the judges to give a rating.  So if some people are close together from a rating one or two percents could make the difference then. And I think the chances would be for all the same, because one will get more in that point and others will get more in other points. ;) (\s/)


Some judges already give scores with up to two digits after the decimal, and I rounded them to the nearest tenth (effectively being out of 100).

As for your proposed scoring categories, I already decided it was wrong when I read "story". I feel as though you just unpacked the already existing categories and threw in some elements which are way too subjective. Desired difficulty is different for every person, and isn't even a measure of how good a game is in the first place. There's no reason difficulty should be its own category when an extreme difficulty could result in a lower playabality score, perhaps, and a game with no difficulty could lose points in entertainment, for example.
Story can never be a category on its own for judging purposes. Some games don't have stories. On purpose. And they're perfectly fine games. In fact, one of the five games I consider to be perfect has exactly no story, and all of the others convey story through gameplay and little through dialogue. It's silly to score a game on how "good" its story is, when there's so much more to story in games than just the plot (and also it's all incredibly subjective).
The way you've described graphics and sound make it seem as though an Atmosphere category could cover those elements and more. However, again, treatment to atmosphere is subjective, so we bundle graphics and sound in to a more concise and objective category, Polish.
Bug Density is a factor of Polish and Playability. There are more things that can break a game than bugs, though, so we have to take them in to consideration as well, which is why it's not its own category.

All this is fine for reviewing games, but not judging them in a contest. It seems a bit basic, arbitrary and dangerously subjective. We try to bundle distinct categories that can include whatever aspects the game does or does not have. This is why we have broader terms than one might expect - to be all-encompassing but also concise. It's tricky, but you just need to forget what you think is important in a game and think about the experience itself.

But I can no longer design games for myself. I have to design for the judges now. If I know a lot of the judges will swoon over an amazing epic story, I'd be stupid not to do that. If I know they hate action games, I'd be stupid to make one.

Also, balance, as I said in the category, is not "This is too hard, F-". It's, "do I have the information necessary to complete the challenge". If you get to a boss that requires put them in a head lock and use a healing item on their neck to KILL THEM, (METAL GEAR SOLID 4 WTF) that's bad balance. There's no reason a player would think to do that. Ever. There isn't a human on this planet that didn't look up how to beat that boss. If the first boss is stupid hard and the next boss is stupid easy, that's bad balance. If the first boss is super easy but the next challenge requires a greek god to imbue their powers to you to handle how hard it is, that's bad balance. It's mostly about consistency.

Where did you read story? I just Ctrl F'ed my post and it didn't show up once.

The problem I have with the current rating system is that some aspects of a game are in multiple categories while some are only in one. Story is in two categories. Bugs affect almost 3.

Quote
There's no reason difficulty should be its own category when an extreme difficulty could result in a lower playabality score, perhaps, and a game with no difficulty could lose points in entertainment, for example.

Less difficulty doesn't make a game less fun. More difficulty doesn't make a game less playable. If you enjoy hard games (Braid, Dark Souls), a fair but difficult game is super engaging to you. Dying is fun because you know the challenge is worthy of your time to complete. If you enjoy easy games (Farmville, Beyond Two Souls, Heavy Rain, Indigo Prophecy), then a game where you never die is actually more engaging to you. You like owning everything and possibly like story more and the gameplay is more about being in their shoes than anything else. What is not okay (and what the category judges) is if you are consistent in your design of that difficulty. A dark souls player is super disappointed when they beat a really hard boss and move on to the next challenge, super excited to be challenged again, only to find they beat the boss the first try. A casual player is disappointed when they are having a lovely time blowing up countless enemies with no difficulty, only to face a boss that is a big jerk and wrecks them for hours until they give up frustrated.

*
Last Stop
Rep:
Level 88
Everyone Off
Secret Santa 2013 ParticipantFor taking arms in the name of your breakfast.Secret Santa 2012 ParticipantSilver - GIAW 10Silver - GIAW 92011 Biggest Drama WhoreBronze - GIAW HalloweenGold - Game In A Week VII
If you get to a boss that requires put them in a head lock and use a healing item on their neck to KILL THEM, (METAL GEAR SOLID 4 WTF) that's bad balance. There's no reason a player would think to do that. Ever. There isn't a human on this planet that didn't look up how to beat that boss.

it was a puzzle in the same way Psycho Mantis in 1 was. ALSO THEY FUCKING TELL YOU WHAT TO DO IF YOU'RE NOT GETTING IT THE FIRST TIME VIA CODEC. lrn2mgs



the thing with the current categories is that it does leave room for some interpretation but it works for any game. with having a "GAMEPLAY" category some judges would judge a game with no combat or something as having a low gameplay score. If we used the same judges every time that's fine as long as we all understood what gameplay meant. when anyone can sign up to judge we need to have categories that work no matter what you're playing.

*
Rep:
Level 87
If you get to a boss that requires put them in a head lock and use a healing item on their neck to KILL THEM, (METAL GEAR SOLID 4 WTF) that's bad balance. There's no reason a player would think to do that. Ever. There isn't a human on this planet that didn't look up how to beat that boss.

it was a puzzle in the same way Psycho Mantis in 1 was. ALSO THEY FUCKING TELL YOU WHAT TO DO IF YOU'RE NOT GETTING IT THE FIRST TIME VIA CODEC. lrn2mgs



the thing with the current categories is that it does leave room for some interpretation but it works for any game. with having a "GAMEPLAY" category some judges would judge a game with no combat or something as having a low gameplay score. If we used the same judges every time that's fine as long as we all understood what gameplay meant. when anyone can sign up to judge we need to have categories that work no matter what you're playing.

That's part of the category tho. Gameplay is anything the player does. There doesn't need to be combat for that. You don't even need a moving sprite for that. As long as that's explained to the judges, it's fine. I really think the judges should be a select few that talk to each other, not a random bunch that don't.

Current categories. Current. I can definitely read.

I just don't like the fact that if I have amazing gameplay but my story is boring and makes no sense, I could get a 5 for entertainment.
« Last Edit: March 20, 2014, 05:15:27 PM by Legacy »

********
Hungry
Rep:
Level 96
Mawbeast
2013 Best ArtistParticipant - GIAW 11Secret Santa 2013 ParticipantFor the great victory in the Breakfast War.2012 Best Game Creator (Non-RM Programs)~Bronze - GIAW 9Project of the Month winner for August 2008Project of the Month winner for December 20092011 Best Game Creator (Non RM)Gold - GIAW Halloween
I just seriously can't agree with a judging criteria that assumes that a game needs to have some aspects.  I've played amazing games without sounds, without graphics, without stories, without gameplay, not all at once but you see what I'm getting at here?

But that's the beauty of it. It doesn't say "Is the music super awesome and have guitars" it says "Does it feel right". Sometimes, silence is exactly what builds up to something perfectly. Sometimes having no graphics at all to play as a blind man is more engaging than having graphics. Games without stories, such as Super Smash Bros, work great. Maybe you play as a deaf guy and can't hear the sounds at all. There's a few ways I could think that would be interesting. There are a ton of times when I wish the designers just shut off the music for a certain scene. That is part of polish. Knowing when to use something and when not to.

I think it's a bad idea to write judgement criteria that assumes certain aspects will be part of the game, then go and say "but it doesn't have to have them to get a high score."  You aren't solving the clarity issue at all.

Gameplay is at the core of games. You also have to remember that as long as the player has control, there is gameplay. Choosing a story path is gameplay. As we've seen with a lot of story based games like Heavy Rain and Mass Effect, that can be done amazingly. Heavy Rain even does some amazing work with quick time events and Balance can easily be applied to that game's core mechanics. Even if the game has nothing but dialogue choices, the category is about making informed decisions. Do the dialogue options give you the information necessary to make the right choices or more broadly, the choices you want? If your game is 100% out of the player's control, you've created a movie.

I agree, however as a judging category "gameplay" falls flat, as judges will likely interpret it as "how fun was the gameplay" or "how broken was the gameplay" and I can easily see it being turned into a subjective category where less gameplay-focused games get penalized.

What about this?:

Balance - Was the difficulty curve fair to a player just starting the game? Did it present challenges that made no sense in the context given? If you died, were you laughing and smiling or frustrated? This is not "omg this game is too hard", it's "Is this game providing me with the knowledge necessary to beat the challenges it presents?" Some games are hard. Some people don't like hard games. This doesn't make the game worse.

Balance is ENTIRELY SUBJECTIVE.

Polish - Was the map design strong? Was the music fitting? Where there sound effects in the right places? This is not "Omg they used all default rpg maker stuff burn him" it's "Does it work? Does it set a mood properly? Does it feel right? Does this piece of music make sense for this situation? Do the sound effects make the scene more believable?"

Bugginess - Were there textures you could walk through? Did you get any error messages? Did the game stall at any points?

Bugs hinder polish, they make the game LOOK less polished, less professional.  I really don't see the reason to separate it out.  Mapping Errors are bugs.  Should these affect two categories or one?

Mechanics - Are the mechanics being used in interesting ways? Are they repeated over and over again? This is not "Have I seen this mechanic before". For example, if the mechanic is smacking things, do you always smack the same things? Or do you start smacking new things that make it interesting?

Conveyance - Do you have a decent idea of where to go at all times? Is it clear what everything does? Was there ever a point where you said "wtf do I do?" or "why is that happening...?" When you fail/succeed, do you understand why and have a decent idea of what you could have done better?

Build Up - Is there a feeling that the game escalates? Is the ending satisfying? Were the way encounters (of any kind) introduced powerful in their tone? This is not, "omg explosions this game is epic" this is, if the game has a dark mood, did the build up to the Shadow Fiend at the end of the game feel right. If the game is a dating sim and you are trying to go out with the super hunk from your school, did the build up to the date feel right. Overall, this is, "did the game make you feel engaged or excited at some point during the game?" If a boss just comes out of no where, that's not build up. If a boss is introduced before hand, teases you a bit and finally kills your best friend at the top of the mountain and you have to solo two of them, now we have build up.

I disagree with all 3 of these proposed categories.  Again, you're pinpointing against what -you- think a game needs to have, and you can't solve the clarity issue by making it more complex.

In fact, your categories barely even take game content into account.  What the game is, is just as important as how well it was executed.  Judgement criteria should be flexible enough to take EVERYTHING into equal account.

I agree with what Pac said.

FCF3a A+ C- D H- M P+ R T W- Z- Sf RLCT a cmn+++ d++ e++ f h+++ iw+++ j+ p sf+
Follow my project: MBlok | Find me on: Bandcamp | Twitter | Patreon

********
Hungry
Rep:
Level 96
Mawbeast
2013 Best ArtistParticipant - GIAW 11Secret Santa 2013 ParticipantFor the great victory in the Breakfast War.2012 Best Game Creator (Non-RM Programs)~Bronze - GIAW 9Project of the Month winner for August 2008Project of the Month winner for December 20092011 Best Game Creator (Non RM)Gold - GIAW Halloween
Double post because I can.

I just don't like the fact that if I have amazing gameplay but my story is boring and makes no sense, I could get a 5 for entertainment.

If your gameplay is amazing, you're going to get a high entertainment score from judges that were entertained by your gameplay.  Some judges may value the story more than others and you won't get as high a score with them, THIS IS FINE.  This is why we use a panel of judges and average their scores.

What is the point of having more than one judge if we're trying to literally remove the entire human element from judgement?

Additionally, YOU'RE IN A JUDGED CONTEST.  Of course you're making the game to impress the judges.  That's the whole point of a judged competition to begin with!  Competitions aren't about giving everybody 10's because they tried, it's about COMPETING, for fun or otherwise.
« Last Edit: March 20, 2014, 05:46:42 PM by tSwitch »

FCF3a A+ C- D H- M P+ R T W- Z- Sf RLCT a cmn+++ d++ e++ f h+++ iw+++ j+ p sf+
Follow my project: MBlok | Find me on: Bandcamp | Twitter | Patreon

*
Rep:
Level 72
~Few people understand the beauty of the night~
2014 Best Topic
Notice that there is no "fun" category. Fun is completely 100% subjective. So is difficulty and originality. These above criteria are things that are much more objective and could be decided by a few judges rather than a bunch, which inevitably causes judging to take a long time. Also, none of the criteria overlap, so failing at once aspect doesn't deplete points from multiple areas. For example, having a buggy game in this competition takes away points from two or more categories.

Although fun may be subjective, a good game developer would be able to make a game fun for players of all types.... So in reality, fun/entertainment is a HUGE part of a good score... in my opinion at least...
Download http://a.tumblr.com/tumblr_lm5v281q6E1qde50fo1.mp3

*
Rep:
Level 87

I think it's a bad idea to write judgement criteria that assumes certain aspects will be part of the game, then go and say "but it doesn't have to have them to get a high score."  You aren't solving the clarity issue at all.


That's not what I'm saying >_< I'm starting to think you're purposely not understanding me. I am saying, if it makes sense to have no music in a certain situation, that that's great! if it makes sense to not have a sound effect, then that's great. But not having sound effects for no reason isn't great. Not having music and having a battle fall flat because of it is BAD polish. Having a really awkward moment where you cut the music to increase the awkwardness is GOOD. It's "does it work".

I am assuming that a game in this contest has gameplay, fitting visuals and fitting sound. That's it. You don't need a story. You don't need super awesome art maps. The visuals and sound don't even need to be the best in the world and written by Nobuo himself, they just need to fit.

Quote

Balance is ENTIRELY SUBJECTIVE.


If all you are going to read and interpret is the title of the category, then yes, balance is entirely subjective. But if you read the things I've been saying, it's more than "is this too hard?" It's is this hard enough for the context we've been given and the previous encounters? If the first boss is silly hard, that's fine, as long as there is a logical way to beat it. What would not be okay is if the next boss is stupid easy for no reason. You have to use common sense with this. If there is a good reason to break a "rule" here, then it's fine. If there isn't, then it's not fine.

Quote

Bugs hinder polish, they make the game LOOK less polished, less professional.  I really don't see the reason to separate it out.  Mapping Errors are bugs.  Should these affect two categories or one?

Yes, they do. But there's no reason you should lose points in two categories simply because there are bugs. Map errors are not bugs, they are map errors. They go in polish, not bugginess. Bugginess is purely: Did I get an error message or did the game completely stop working.

Quote

I disagree with all 3 of these proposed categories.  Again, you're pinpointing against what -you- think a game needs to have, and you can't solve the clarity issue by making it more complex.

In fact, your categories barely even take game content into account.  What the game is, is just as important as how well it was executed.  Judgement criteria should be flexible enough to take EVERYTHING into equal account.

I agree with what Pac said.

Why do you disagree? Also, what content in a game am I not including with the balance, the mechanics, the polish (which includes art, music and SFX), build up...? That's pretty much everything in a game in a nut shell. What do you mean by "what the game is?" Genre?

Although fun may be subjective, a good game developer would be able to make a game fun for players of all types.... So in reality, fun/entertainment is a HUGE part of a good score... in my opinion at least...

I hate RTS games. How do you make that fun for me? I don't like visual novels or games that are too easy (Beyond Two Souls). How do you make that fun for me? Furthermore, how do you do that in one week? Adding difficulty levels in a week is nigh impossible to balance correctly and to basically require it is extremely hard. A good game developer understands that their game has a market audience and that pandering to too many audiences is a good way to make a bad game or a game that is way overscoped.

Additionally, YOU'RE IN A JUDGED CONTEST.  Of course you're making the game to impress the judges.  That's the whole point of a judged competition to begin with!  Competitions aren't about giving everybody 10's because they tried, it's about COMPETING, for fun or otherwise.

In the Olympics, Gymnasts don't lose because the judge isn't a fan of a certain move. They judge based on how well executed it was. There are ways to judge execution, rather than personal enjoyment.

Quote
If your gameplay is amazing, you're going to get a high entertainment score from judges that were entertained by your gameplay.  Some judges may value the story more than others and you won't get as high a score with them, THIS IS FINE.  This is why we use a panel of judges and average their scores.

This assumes there are always an equal amount of judges who are in the game's target audience and an equal amount who are not. This is almost impossible. If there's even a slight imbalance in the number of people who hate your game's genre, you are automatically at a disadvantage.
« Last Edit: March 20, 2014, 06:25:37 PM by Legacy »

********
Hungry
Rep:
Level 96
Mawbeast
2013 Best ArtistParticipant - GIAW 11Secret Santa 2013 ParticipantFor the great victory in the Breakfast War.2012 Best Game Creator (Non-RM Programs)~Bronze - GIAW 9Project of the Month winner for August 2008Project of the Month winner for December 20092011 Best Game Creator (Non RM)Gold - GIAW Halloween
If all you are going to read and interpret is the title of the category, then yes, balance is entirely subjective. But if you read the thingzs I've been saying, it's more than "is this too hard?"It's is this hard enough for the context we've been given and the previous encounters? If the first boss is silly hard, that's fine, as long as there is a logical way to beat it. What would not be okay is if the next boss is stupid easy for no reason. You have to use common sense with this. If there is a good reason to break a "rule" here, then it's fine. If there isn't, then it's not fine.

But the thing is...ALL OF THIS is subjective.  I may say that ATLUS games aren't too hard, but you might say they are.  It doesn't matter that it is the same balance curve, we have different perspectives.  ADDITIONALLY not all game types even rely on balance to begin with!  Why write categories that don't apply to all game types to begin with?

Yes, they do. But there's no reason you should lose points in two categories simply because there are bugs. Map errors are not bugs, they are map errors. They go in polish, not bugginess. Bugginess is purely: Did I get an error message or did the game completely stop working.

A bug is anything within the game that isn't how it is intended to be.

Why do you disagree? Also, what content in a game am I not including with the balance, the mechanics, the polish (which includes art, music and SFX), build up...? That's pretty much everything in a game in a nut shell. What do you mean by "what the game is?" Genre?

Because I don't think we need to separate every aspect of the game out and grade them separately.  A game is the sum of its parts, we should grade it as such, not by the quality of the individual moving pieces.

In the Olympics, Gymnasts don't lose because the judge isn't a fan of a certain move. They judge based on how well executed it was. There are ways to judge execution, rather than personal enjoyment.

In the Olympics, judges aren't actively participating in the media that they are judging.  They are simply observing.  Games are interactive, and meant to convey something to the player, be it a message or simply entertainment.  If a game is boring and doesn't grip the judge by their collar and drag them into the game world, then it should lose points.

This assumes there are always an equal amount of judges who are in the game's target audience and an equal amount who are not. This is almost impossible. If there's even a slight imbalance in the number of people who hate your game's genre, you are automatically at a disadvantage.

If you make a game that isn't fun to half the judges, or isn't fun to most of the judges, it probably isn't very fun, or you knowingly made a very niche title.  Sorry, but that's how aggregate statistics work.

---

You are creating arbitrary limitations based on your own opinions.  As I keep saying, judgement criteria needs to fit any genre, any graphical style, etc... and our system already appears to accomplish this. 

Polish covers how well the art meshes ( including visuals, audio, dialogue, etc... ), as well as if there are any issues with presentation that disrupt the look and feel of the game.
Playability covers balance, gameplay crashes, issues with progression, etc... 
Ingenuity rewards the creator for putting effort into making their game unique.
Entertainment ( or 'Immersion' if you prefer ) gauges how much the game pulled in the judges and kept their interest.

All games need to be polished, playable, clever ( some more than others ), and they need to capture your attention.

I would challenge you to find a genre or game that could not be evaluated based on these principles.

FCF3a A+ C- D H- M P+ R T W- Z- Sf RLCT a cmn+++ d++ e++ f h+++ iw+++ j+ p sf+
Follow my project: MBlok | Find me on: Bandcamp | Twitter | Patreon

*
<o>_<o>
Rep:
Level 72
2014 Best Artist2013 Best ArtistFor taking arms in the name of your breakfast.Gold - GIAW 10Contestant - GIAW 9
Now THIS is a GIAW!
Looks at that creative discussion!

*
Rep:
Level 72
~Few people understand the beauty of the night~
2014 Best Topic
All games need to be polished, playable, clever ( some more than others ), and they need to capture your attention.

I would challenge you to find a genre or game that could not be evaluated based on these principles.

Well said.
Download http://a.tumblr.com/tumblr_lm5v281q6E1qde50fo1.mp3

*
The Hero of Rhyme
Rep:
Level 83
( ͡° ͜ʖ ͡°)
Project of the Year 20142014 Best RPG Maker User - Story2014 Queen of RMRK2011 Best Newbie2014 Best RPG Maker User - Creativity2014 Kindest Member2013 Queen of RMRKBronze SS AuthorBronze Writing ReviewerSecret Santa 2013 ParticipantFor taking arms in the name of your breakfast.GOOD!For frequently finding and reporting spam and spam bots2012 Best Yuyubabe Smiley2012 Best RPG Maker User (Creativity);o
Okay, so I know I'm not exactly the epitome of understanding and kindness lately, but I don't quite think this is the right approach to encourage new ideas around the forums.

Either way, I can see where you're both coming from at this point. But I don't think its quite fair to assume that Legacy is basing those points off of his opinions, per se... I think a lot of them are very good and can at least be used as subcategories for the guidelines. On that note: adding/defining those subcategories might be helpful. Anything that we agree should be important, but maybe not -too- important, could be a subcategory. We could have each of those scored out of 10 total points, and then take the average of those (or tally them up) as that general guideline's points. And so on, for each guideline.

Spoiler for Example:
Example: "Polish" could be "Visuals" "Audio" and "Story/Writing/Whatever". If we encounter a game with no writing, we can always just average out the other two and use that for that game's general score for "Polish". Or something. I dunno, it's an idea.


Assuming the person made a game with story elements:

Visuals - 6.4
Audio - 5.3
Story/Writing - 4.1
Polish Average = 5.27

Assuming the person made a game without story elements:

Visuals - 6.4
Audio - 5.3
Polish Average = 5.85

I understand that there's probably some flaws with this system, since it may - or may not! - give an advantage to those that are choosing to include story elements. Most games will have them, either way, but it's entirely up to the person making the game. Just a thought I figured I'd throw out there. :drsword:
« Last Edit: March 21, 2014, 12:32:05 AM by Queen yuyubabe »
Spoiler for My Games and Art:
ℒℴѵℯ❤


My Artwork Thread

The Lhuvia Tales [Current]

Ambassador [Complete]

The Postman [Complete]

The Wyvern [Complete]

Phoenix Wright: Haunted Turnabout [Complete]

Major Arcana [Cancelled]


*
my name is Timothy what's yours
Rep:
Level 79
Hello
2014 Most Missed Member2014 Zero to Hero2014 Best IRC Quote2012 Zero To HeroSecret Santa 2012 ParticipantContestant - GIAW 9For frequently finding and reporting spam and spam bots2011 Zero to Hero
I just think that kind of system is unnecessarily overcomplicated. We have 4 succinct, all-encompassing categories that should be easy to understand.
it's like a metaphor or something i don't know

****
Rep:
Level 43
Somewhat got a project? (\ô/)
GIAW 14: ParticipantParticipant - GIAW 11
But that's the beauty of it. It doesn't say "Is the music super awesome and have guitars" it says "Does it feel right". Sometimes, silence is exactly what builds up to something perfectly. Sometimes having no graphics at all to play as a blind man is more engaging than having graphics. Games without stories, such as Super Smash Bros, work great. Maybe you play as a deaf guy and can't hear the sounds at all. There's a few ways I could think that would be interesting. There are a ton of times when I wish the designers just shut off the music for a certain scene. That is part of polish. Knowing when to use something and when not to.
That was what I tried to say, is the graphic fitting, means in example of MLP, there are only less colors, kinda like 4-16 colors at once on screen. That would be about a high score. But if a horror game has bright and colorful presentation it would be a low score. I meant the points kinda like how much they fitting.
For sound it is kinda like you said, if someone is blind, it would be great, that you need to listen to the sound to get to dungeons. So it's about fitting the situation. (\s/)

The problem I have with the current rating system is that some aspects of a game are in multiple categories while some are only in one. Story is in two categories. Bugs affect almost 3.
I think the same, it's kinda unfair. You are doing one mistake and you get a low score in allmost every point. :/
I think if it is clearly listed it would be a fairer competition. Because someone who didn't made a story get's a 0 there, but maybe he get's a 9.2 or higher for the inventiveness. So everyone has a chance and not only the creme del a creme. I guess a beginner may lose many points in bugs, graphics, sound, but may get a high score on story and gameplay. I guess the review way gives more people a fair chance and not only the pro's. ;) (\s/)

By the way, even with the momentary judgement system belongs all to the result of all judgement scores. So there would be not much difference to know. ;) /)

*
my name is Timothy what's yours
Rep:
Level 79
Hello
2014 Most Missed Member2014 Zero to Hero2014 Best IRC Quote2012 Zero To HeroSecret Santa 2012 ParticipantContestant - GIAW 9For frequently finding and reporting spam and spam bots2011 Zero to Hero
Because someone who didn't made a story get's a 0 there, but maybe he get's a 9.2 or higher for the inventiveness.
No. That's exactly why things like "story" can't be a category.
it's like a metaphor or something i don't know

*
Rep:
Level 97
Definitely better than Hitler.
2014 Best Musician2014 Best IRC Chatterbox2013 Funniest Member2013 Best Musician2013 King of RMRK2013 Best Use of Avatar and Signature SpaceFor the great victory in the Breakfast War.2012 Best Username2012 Best MusicianFor frequent good quality Wiki writing [citation needed]Most entertaining member on the IRC2011 Best Musician2011 Funniest Member2010 Most Missed Member
I just want to say that this discussion is dumb. Now having said that:

1. Some of you are exaggerating the meaning behind what the other is saying to make it seem like what you're saying is more reasonable.
2. No scoring system is perfect, but what's being proposed is less perfect than what is already in place. It's a simple contest, the scoring criteria should be simple.
3. If the way games were scored was really in need of being changed, it probably would have been brought up well before GIAW reached double digits.
4. Scoring story in both Entertainment and Ingenuity is dumb, that should actually be changed.
5. Everybody shut up.
:tinysmile:

*
Rep:
Level 102
2014 Best Non-RM Creator2014 Biggest Narcissist Award2014 Biggest Forum Potato2013 Best IRC Chatterbox2013 Best Game Creator (Non-RM)Participant - GIAW 112012 Most Successful Troll2012 Funniest Member2012 Best Use Of Avatar and Signature space2012 Best IRC ChatterboxSecret Santa 2012 ParticipantProject of the Month winner for November 2009For being a noted contributor to the RMRK Wiki2010 Biggest Forum Couch Potato2010 Best IRC Chatterbox2010 Most Successful Troll
I do think the scoring system system is in need of a change. Personally, here's the categories I'd have:

Spoiler for:
Story
Game feel
Backstory
Lore
Ambiance
Atmosphere
Cutscene quality
Cutscene length
Subtitle font
Jumping physics
Music melodies
Music catchyness
Music chord progression
Music wind-chime sample quality
Voice acting (male)
Voice acting (female)
Voice acting (transgendered)
Voice acting (genderless)
Voice acting (non-humans)

*
The Hero of Rhyme
Rep:
Level 83
( ͡° ͜ʖ ͡°)
Project of the Year 20142014 Best RPG Maker User - Story2014 Queen of RMRK2011 Best Newbie2014 Best RPG Maker User - Creativity2014 Kindest Member2013 Queen of RMRKBronze SS AuthorBronze Writing ReviewerSecret Santa 2013 ParticipantFor taking arms in the name of your breakfast.GOOD!For frequently finding and reporting spam and spam bots2012 Best Yuyubabe Smiley2012 Best RPG Maker User (Creativity);o
I SHALL NOT BE SILENCED (because I already typed this out and don't want to have wasted my time ok)

I just think that kind of system is unnecessarily overcomplicated. We have 4 succinct, all-encompassing categories that should be easy to understand.

Errr... Polish, maybe. But it says "visuals and audio" and out of nowhere, it seems like it includes other stuff that has to do with polish (whereas I took it literally as nothing but visuals and audio).

Gameplay is typically a good guideline, but its very, very broad.

Ingenuity is the most confusing one. I think it should be split up or watered down or just explained better.

Entertainment is a tough call, though. The other guidelines are going to have influence over it, such as a game with many bugs being scored lowly on this because it annoyed the judge and ruined their fun. Also, this is probably the worst genre for constructive criticism. When the judges say "you're game ain't fun," that's probably the most likely thing to keep me from actually finishing the game. Especially because most games aren't as fun in their beta phases.


I think the use of subcategories will outline a basic list of what each judge should look for. Not only that, but it might actually help out the maker, too. They just spent a majority of their week making a game and pouring their heart into it, and many of them wish to continue making or improving their games. For example (using Polish again), it might really help them to see: Visuals - 5.5/10. Audio - 7.9/10. Whatever else - 9,000+/10.

If everything is broken down, it'll be much easier to see what can be improved, while also not putting too much pressure on the judges to have to write comments for everything (though comments are always helpful).

Spoiler for More Example whee:

Polish:

Visuals - ?/10
Audio - ?/10
Writing - ?/10


Gameplay:

Playability - ?/10
Mechanics - ?/10
Balance - ?/10


Ingenuity

Freshness :mad: - ?/10
Entertainment - ?/10
Direction - ?/10



(I just filled in randomly somewhat suiting stuff. The contents would require further debate, considering if anyone likes this.)
Spoiler for My Games and Art:
ℒℴѵℯ❤


My Artwork Thread

The Lhuvia Tales [Current]

Ambassador [Complete]

The Postman [Complete]

The Wyvern [Complete]

Phoenix Wright: Haunted Turnabout [Complete]

Major Arcana [Cancelled]


*
Rep:
Level 97
Definitely better than Hitler.
2014 Best Musician2014 Best IRC Chatterbox2013 Funniest Member2013 Best Musician2013 King of RMRK2013 Best Use of Avatar and Signature SpaceFor the great victory in the Breakfast War.2012 Best Username2012 Best MusicianFor frequent good quality Wiki writing [citation needed]Most entertaining member on the IRC2011 Best Musician2011 Funniest Member2010 Most Missed Member
I do think the scoring system system is in need of a change. Personally, here's the categories I'd have:

Spoiler for:
Story
Game feel
Backstory
Lore
Ambiance
Atmosphere
Cutscene quality
Cutscene length
Subtitle font
Jumping physics
Music melodies
Music catchyness
Music chord progression
Music wind-chime sample quality
Voice acting (male)
Voice acting (female)
Voice acting (transgendered)
Voice acting (genderless)
Voice acting (non-humans)

Whoa, ambiance AND atmosphere? That's just silly. Narrow it down to just one though and I think you're on to something.
:tinysmile: