This is a bit rant-ish, but what the hell. I read a fair amount of game books, and I also write game books, and I use game books fairly often, too. But I also come at them with a critical eye towards making things better or more useful. Or, at least, de-cluttering them.
Tell me what you are, not what you are not.
A rulebook should tell me what the game is and how to play it, and not waste words telling how the game isn't.
If you feel compelled to tell me how different your game is from other games, do it in the intro or do it somewhere else, like a blog post or free ebook. Embedding it into the book and making me read it makes me feel tired all over. I feel like I paid you money to have you tell me what you're not providing to me.
This is especially true when I read some OSR stuff, because commentary on "what this game doesn't do" is aimed at 3x and 4x D&D. Now just imagine you're someone who never played 3x or 4x D&D - do you want or need to be told what the game isn't? If you write "this game does X, unlike how some other games do it" just think about how someone who doesn't play those other games will read it. "We don't provide X gold for monsters of Y level just because." Okay, so that happens in 3x D&D, then? I come from GURPS and AD&D and Rolemaster and I have no idea what the hell you're talking about.
Remember that game writing is technical writing, and I'm going to use this book as an inspiration for game and a reference in play. That's all it needs to be, don't spend wordcount on explaining what it's not doing.
Show what to do, not what not to do.
I have to admit I like Swords & Wizardry a lot, but there is this line in S&W Complete that made me roll my eyes:
"Russell: “Check for traps.”
Referee: “It doesn’t work like that. You can check a small mechanism
to see if there’s a trap in it, but you can’t just illuminate the place with
find traps radar.”"
Okay, now I know how the game isn't played, hurrah. This isn't very helpful from a technical, use-the-book-to-play sense. It could have skipped right to "I look for tripwires" and shown me how it is expected to be done. Show me the right way and skip over the wrong way. Lead by example and don't include the bad examples.
Don't tell me how much the stuff I like sucks.
Kevin: "Do you have the new Depeche Mode record?"
Bruce: "Yeah it's over there, but it sucks."
Kevin: "Well, do you have the latest Pixies album?"
Bruce: "Yeah, but it sucks. All that new stuff sucks"
- Kids in the Hall, The Doors Sketch
I don't like it when a book's author, or editor, or designer, or whoever, tells me about how bad other stuff is. Sometimes, I like that other stuff, and that other stuff is why I'm here reading your take on the same or similar thing.
I have a nice set of Conan books that compile R. E. Howard's original stories in their original form - as typed, pre-edit, pre-publication, finished or not. It's a really cool series. But the first book's intro heavily implies (without naming either) that L. Sprague de Camp and Lin Carter's Conan stuff is a cheap ripoff. Maybe it is, maybe it isn't. But I loved those books. They're the first ones I read. I like Howard's original stuff better, but I bought those collections of the originals because of the interest inspired by the later renditions of the work. Instead of saying "You might have loved those, but here is the original stuff - and I think you'll find it equally interesting or even superior" it instead comes across as "Liking those was bad, they suck."
Don't do that. It's bad form to insult your audience, and you are putting people who like what you have and like what you don't on the defensive. Sometimes I see this in RPGs - it's a whole "unlike in other games, this game does it right" attitude. You hear it in the "new school games with a sense of entitlement" rants. Telling someone who loves D&D 3.5 or Pathfinder that their game system's underlying assumptions are inherently bad is going to be a turn off. You're not going to win people over with "the game you like sucks."
It can be more subtle, too, when you tell people their play style sucks. GURPS did this, too. The 3e Basic Set discussed simple hack-and-slash gaming in a sidebar which ended with
"If you want to play a more "mature" level, and create a situation that actually makes sense, you have advanced to to level of adventure design. Congratulations, read on . . . "
Gee, thanks, now I feel great about wanting to have fun with my friends the immature way. You know, fighting monsters and taking their stuff in a bunch of tunnels full of 'em. GURPS 4e kept that line, but with a subtle-but-important tweak:
"If you want to create a situation that actually makes sense, you have advanced to the level of adventure design. Congratulations. Read on . . .
See the difference? One makes me feel like a jerk for wanting to kill monsters for their stuff, the other makes me feel like the rest of the chapter is there if I need it for more developed rationales and more verisimilitude in my games. Nothing is gained and a lot is lost with that extra snarky comment.
Think of this when you talk about other games to gamers who play them. Do you want those gamers to love the kind of games you do, too, or love them instead of the games they love now? Which is more important?
Look at it this way - Moldvay D&D doesn't come out and tell you AD&D sucks, or Pendragon sucks, or whatever else sucks. It comes out and tells what D&D is, that you can have basically unlimited imaginary fun with it, and here is how to play it. Labyrinth Lord does the same - it just plunges straight into it, taking as read that you're interested and want to learn the game without sarky asides at different games.
If your game really is inherently better, you won't need to tell me. I'll notice.
If you've told me once, it's enough.
You really don't need to beat stuff into my head. Tell me once, in one place, and refer back to it. Don't keep repeating yourself. By the second time I read something I'm checking to see if it's the same as what you wrote the first time. By the third I'm wondering why you think I need to hear it again.
If it needs to be in multiple places, fine, but use page references to point back and forth between instances of the same thing. Even then, ask yourself, do I need to repeat this rule, this example, or this suggestion again? If you are even a little bit unsure, cut it.
Some of us read and check references and remember what you write. So try to be consistent. I recently dinged DCC for this in my review. It harped on how illogical heaps of coins for treasure and sages getting paid in gold for knowledge, but said to pick any treasure system you want (all of which provide heaps of coins for treasure) and then listed sage prices in coins.
Mistakes happen, but so does playtesting and peer review and proofreading. Little bits will get through but you're going to want to ensure the system as a whole makes sense. If you have a consistent message in the writing, it will show through. Consistent doesn't mean rigid, of course. You can stay modular as long as modularity is part of the message. S&W does a pretty good job of this - it provides support for Ascending AC and Descending AC, different Saving Throw breakdowns, and different initiative systems, without any issues I can see. Why? Because it presents them equally and explains how to do them and says, pick one, and trusts the GM to do so. GURPS thrives on this, and is so Rule Zero heavy that you really can't get started until the GM briefs you on what's used and not used.
I'm sure some RPG designers, and readers, have different opinions on this. But the above is stuff I look for, and I notice. You can really sum most of it up as "Be positive and just show people where the fun is. Don't insult them. And proofread for inconsistencies."