How Can We Judge an RPG?

So, I've been running into a very interesting attitude of late due to all the game reviews we've been doing, best summarized as the following.

"The game is fun if you ignore the setting and change the rules."

I'll let that sink in a moment.

It's a true statement, in that if you make a game into another game that you like, you will like it. It's also an awfully idiotic statement in whether or not this evaluates a game as good.

What can we rate an RPG on and how can we justify it? Games run different ways- some are deadly, some are forgiving, some are detailed and slow, while others are lite and fast. To say which approach is right is sort of a cost benefit analysis combined with an identification if the System matches the Setting.

For a Quick List of Traits of a Good Game, I would Probably List:
-Balance
-System Supporting Setting
-Effort to Play

Balance:

Balance means fairness, generally an equal footing with the player standing next to me at character creation. If we're playing a squad of monsters in a game meant for it, my werewolf should be equally useful to his mummy. The paragon of game balance is the HERO system. Years of tinkering and toiling have been spent to try and create a system in which 'you get what you pay for'. Guidelines as to what makes something too powerful, warning stickers on potentially broken powers, and other safeguards and balances work to make HERO one of the most fair systems ever.

There are lots of examples of poor balance out there, but I think one of the easiest is early model RPGs where rolled stats reward people more for rolling well. As if winning wasn't enough, you can win more, so you can win more, before the game even starts.

System Supporting Setting:

Take Seventh Sea, for example. It's a game of swashbuckling adventure, more or less. There are fights, epic virtues and vices, plots based around character, larger than life characters, rules to have awesome duels as well as cut a swath through lesser foes. These are all useful features for a game about this kind of adventure. The rules reinforce the tone and action of the setting and this is what makes it a good game. It has it's rough points, but thats why a lot of games get new editions to sand those out.

Effort to Play:

There is work in playing games- making characters, reading setting, learning rules, rolling dice to get results, etc. The more effort this takes, the more pay off you should get for it. Savage worlds is probably a paragon of bang for your buck gaming: you can make characters quickly, use the book and grab NPCs to run them, and it has a low handling time for rolls made. It's quick to get the basics down and just start playing if you have a GM willing to create a game on the fly.

Hackmaster may be the height of effort to play problems. Big book full of rules, long character gen, complicated charts- we've heard the game is fun, but we've never had the gumption to make it through all the effort required to actually play it.

While more opinion and less quantifiable, I would also List:
-Setting Appeal
-Style
-Engagement

Setting Appeal:

Some people like apples, some like oranges- there is no way to please everyone. However, there are some steps you can take to make a game fun for more people, which means making it accessible to everyone. This means making the setting something that doesn't exclude people based on race, religion, orientation, or gender. Thankfully this is not an issue in most games, even the alternate history ones, which make allowances for those characters who would otherwise be discriminated against in less enlightened times.

Style:

There are little widgets to a system that make it fun. Sometimes it's a single feature, like Cthulhu's sanity system or the Spiritual traits of Riddle of Steel. Sometimes, it's the ability to build each power with the same care you would build a small character, like in HERO. Some games come out so bland you can't really latch on to it or do more than shrug and say 'It works'. I like games that have a bit of quirk that fits the system and does something cool.

Engagement:

A good system should keep you caring about whats going on, keep you engaged. This can vary a great deal between game styles- rule's lite RP intensive systems may accomplish this with a low handling time and fast results, while a more tactically intensive mini's based game may have very heavily engaging combat rules. This goes right along with System Supporting Setting. To summarize, a good combat keeps me on my seat, paying attention and planning what I will do next- a bad combat makes me read a book and wait to roll for my swing at the baddie.