There are three basic qualities I look for in a role-playing game system: Complexity, Simplicity and Flexibility. These are defined as follows:
Complexity defines a game that has depth in its rules. Complexity does not mean being complicated but they can be confused for each other. Complexity goes deeper into defining game elements and refines character concepts. Any game system that adds personality, history and goals to characters can be thought to have Complexity. These elements of course have to be meaningful to other game mechanics. If not they will remain unused and exist for no reason whatsoever.
An example of complexity is in the game GURPS. Everything that is added to the character is used; advantages, disadvantages and skills are all Complex elements that are used regularly during the game session.
Simplicity is probably the easiest of the three criteria to understand. It means the game is made as simple to understand as possible. Rules are streamlined to offer quick and easy play and are kept to a minimum to speed up learning of the game. Idiomatic game rules are also kept to a minimum. Idiomatic rules are any rules that are only found in a particular game system and cannot be related to real-world concepts or observations. Examples abound in most wargames, the most idiomatic gaming systems found anywhere. Wargames must simulate large-scale encounters, so must be simplified to an impractical degree. Impractical, that is for role-playing, because each unit must be represented by a few descriptives and numbers to speed up play.
A good example of Simplicity I have encountered is HârnMaster (the first edition). The rules for the entire game is included in one 146-page rulebook. The rules are complete and well detailed. The game aspect are well managed and play is smooth.
Another aspect of Simplicity is consistent game mechanics. Meaning that all aspects of the game are handled in a similar fashion to eliminate having to learn different rule mechanics for each activity of the game. Is magic, combat, psionics and using a skill handled similarly? If all of these game aspects use the same rules and concepts then the game has taken a step towards Simplicity.
GURPS handles combat and skill use differently and some new rules must be learned for magic and psionics. This game is not rated as a simple system to learn or use. Not when compared to HârnMaster where most of the concepts for magic, combat and psionics are interchangeable.
Flexibility defines a system that can be used with a wide range of play genres and styles. A very flexible game will offer rules for many different elements of play, even if those elements are not used by every player or game-master. Rules to cover psionics, magic, low tech, high tech and different levels of combat from one-on-one to massive armies should be presented. Extremely flexible games are very rare and those that exist tend to suffer from multiple personalities. They rarely know if they are fantasy, sci-fi, cyberpunk, contemporary etc. And they tend to confuse players when they do not know which rules the game-master uses and which ones are not used. They require fine control on the part of the game-master playing them.
But they offer one thing in their favour. When the rules are learned, never will the players and game-master have to learn a whole new set of rules if they want to change style or genre of play. This can be a huge asset when playing many different games or campaigns at the same time of sequentially. For someone who likes variety, this is a boon.
There are several very good Flexible system out there, like GURPS, Hero System and the Storyteller system from White Wolf. Each can be used to play a variety of game styles and campaign genres. Support is very important to make the most of a Flexible system. If someone wants to play several genres at the same time in the same campaign, reference manuals must be available to get the feel and rythm right. But such games invariably require a lot of work from the game-master.
There you have it, my three criteria for a good game system. As you have probably surmised, it is next to impossible for a game system to have high marks in all three aspects and I have found this to be so myself. I have owned, played, tested or read at least four dozen role-playing game systems and very few of them have met all three of my strict criteria. Of all of them I have retained about twenty in my bookcase because they either have fascinated me with various elements or intrigued me for some reasons.
I will eventually rate these retained games according to the above criteria, but not now. In the meantime, if you want to know what I think about a few of the games I have played you can read the individual entries for each game.