There is a very interesting post over on Power Score about PvP in D&D.
I've played in some games, not fantasy games, where PvP was common. Players and their characters had different motivations, different goals, and very different approaches to those goals. Net result? Lots of character death. I personally caused a lot of them. Maybe the most, person for person - I know I killed or had killed at least three characters (two belonging to the same guy, one a drop-in try-out player who left behind an extremely messy, unreliable, plot-disrupting, and dangerous PC.) It didn't generate any bad blood because we went into the game knowing PvP was part of the deal.
I recall my friends talking with some admiration about how one guy really got the point that Vampire wasn't D&D when he got the other vampires killed for his own personal benefit. It was that moment of realizing, hey, wow, we're dead and he's right, we're not a party of friends but a cluster of temporary allies.
But we've lost players, permanently, to PvP issues - even when one of them was driven to PvP by external domination (via Enslave spell, specifically) by an NPC "ally" of the group who realized fracturing them up and killing them was more affordable than paying them. Even with a logical in-game excuse that the player couldn't control (he rolled badly when it mattered, got captured, got Enslaved), it was still enough to damage the group.
For that reason we generally take a founding idea of the game that it is either pro- or anti-PvP. Pro-PvP doesn't mean we have PvP. Our pirates campaign was pro-PvP, but I can't recall any incidents. It's possible a PC shot an NPC or something (everyone had a lot of PCs and player-controlled NPCs), but I don't remember. Had it happened, well, these are pirates - bound to get some internal crew stresses. Anti-PvP games - our default stance - means you don't go against the group. You always find some reason - in game, on your character sheet, or whatever - to not go PvP even if it would result in PvNPC conflict. Sense of Duty is a common one, as are twists on disadvantages that make valuing your allies a means to an end in the game. The guy with Greed might pocket stuff without informing the group, or make deals for money when the group as a whole would prefer something else, but might also justify the group on the grounds that "These guys help me make so much more money that I did on my own!" That kind of stuff.
As a GM, I'll still happily use domination-type powers and possession-type magic against PCs, but I'm much more inclined to make it short term and immediate. My players don't seem to mind beating down the guy who has been mentally dominated by the King of the Vampire-Lich-Trolls or evil god as much as they mind thinking any normal, non-combat interaction over a longer campaign might a PC turned against the party.
PvP can be fun - it's extremely fun in video games, Munchkin, Paranoia (it's not nearly as much fun without PvP), every board game we play except the occasional co-ops, etc. But it comes with enough stress that we leave it off by default. Don't bring an assassin to the party with the paladin, and don't bring a wizard-hater to the wizard-happy party, and don't make a loner for the group game. Not if I'm running it, anyway. It's just easier for everyone to relax and enjoy the game that way, we've found. We're up for PvP but it's got to be a basis for the game.