Matt Riggsby had a nice Daily Illuminator and then a nice blog post about adding rules to GURPS.
What I tend to look for in a new rule:
- Covering new ground. This is pretty hard, because a) the basic rules already cover a huge variety of circumstances directly so you don't need new rules to do so, and b) a lot of the potential special cases are already covered.
- It has to drill down on the detail for something worth drilling down into, or scale up in a way that makes the game faster and more fun. In other words, it has to expand on something that works at a grainier/most abstract scale already or takes a more detailed rule and makes it more grainy/more abstract.
- It has to be 100% compatible with the rules you're mostly likely to pair it up with.
Ideally, you want all three. In fact, if you don't get all three, it probably needs more work. Technical Grappling does this well (covers new ground in terms of details and special cases, drills down the details, 100% compatible with the rest of the system). So does Social Engineering (covers new ground in terms of social skill roll effects, drills down, 100% compatible) and so does Action 2's concept of BAD (covers new ground in abstraction, abstracts some things yet makes them concrete, 100% compatible with the rest of GURPS).
Above all, of course, the rules have to be fun. Fun for the people who want rules for that sort of thing. Detailed rules on weapon length, martial arts styles, and special rules for smacking people's faces into car doors make you grit your teeth and moan about GURPS having too many rules? Take a pass on Martial Arts. Look at Social Engineering and roll your eyes and say "You're supposed to role play!"? Not for you. Look at Action 2 and say "You can't just abstract all the difficulty in taking down the bad guy to a number!"? Also not for you. On the other hand, if you say, "We need to make Goju Ryu different from Shotokan in this game," or "I want some mechanics for social climbing," or "I need a concrete way to explain why Bond doesn't just drive up to Blofeld's house and shoot him," well, you have the rules you need.
That's essentially what I look for in new rules. When I write them, though, I have another criteria: is it possible to do this already with existing rules? If so, I do that first.
When I'm writing rule-heavy books - say, DF 12: Ninja - most of what I write isn't new rules. Most of the opportunities to write those are past at this point. Mostly what I do is demonstrate new things you can do with the existing rules. Nothing in Killing Strike or Weapon Master (Ninja Weapons) is really new-new, it's just showing a new execution of existing concepts. On the other hand, somethings you need more detail. The very light rules for Loyalty in Basic Set are fine, but not when you've got a gaggle of hirelings you're taking into the haunted castle to fight monsters. So a new, expanded rule was needed there.
New rules, house rules, and new executions - in all cases I want fun and compatibility and actual newness of utility. If I can't get rules to do those things, I don't want to use them.