At what point did D&D go off the rails?
D&D never went off the rails. It never jumped the shark. No book, no Gygax rules-as-written-or-it's-not-D&D paragraph, no spell, no monster, no module, no edition. Never.
What happened, instead is this. D&D kept growing, evolving, changing, and moving. Books kept getting written, modules kept getting purchased and used, new players kept joining and old players kept dropping off.
At some point, you and D&D grew together. The stuff that was out was magical to you. The new stuff was the kind of new stuff you wanted, generally (and what you didn't was small enough to ignore.) At some point, though, what you wanted from D&D either was in a different direction than what was coming out (your tastes evolved) or you didn't need anything new (your needs basically froze.) Then, suddenly, D&D had gone wrong.
But it didn't. It just went elsewhere. It changed from what you wanted into something else. Or you changed from what you were to what you became. More likely, both.
D&D 4e? That was the game some people wanted. Same with 1st edition AD&D, the Greyhawk supplement, 2nd edition, those splatbooks, Unearthed Arcana, etc. etc. etc. If it was what you wanted, you'd mark the date of the death of D&D later than someone who didn't want that stuff. It feels downright odd to have people explain how the system went from great to suck before you experienced your most magical moments with the game. It's like having someone say a band you like was good up to album X, and you got into them with album Y. It's actually kind of insulting, too - you didn't show up until it sucked, so therefore you like the bad stuff.
Personally, I abandoned AD&D when 2nd edition was starting to come along and I found Rolemaster and GURPS fit better with what I enjoyed. Unearthed Arcana is much derided but it was the basis of the single greatest AD&D campaign I ever ran. Yet for some D&D went off the rails and became something "other" when the thief showed up. When you and what was out for and coming out for D&D overlapped enough, it was fine. When you and it went different directions, you just diverged. It didn't start to suck, and neither did you. You just took a different fork in the road.
And that's pretty much my opinion on the subject. "Not for me" and "bad" are not the same thing. Things that change really go from "for me" to "not for me" or vice-versa. I really only find the badly-done to be bad (error filled, poorly written, etc.) and the rest is just for me or not for me.