I'm sort of a neutral party in this "debate". I originally started on GT3 when my brother got it for PS2 (I was more into the arcadey racing games before that), got GT4 when it first came out, and got GT5 maybe a few weeks after it was released. I never really got into xbox until this past summer when I saw what all was included in FM3, so I got a used 20gig. I've pre-ordered FM4 LCE, so I'll get that when it's released.
Anywho with that out of the way, and after playing both series, I like to think I have a fair opinion of the two. On the one hand, I love the realism that GT is striving for with the accurate tracks and great Premium models, and I liked how the licenses would unlock what races you could do (rather than the points system in GT5.) They have/had so many cars to choose from which I thought was cool compared to other games which maybe had 200 at most (like GT3, though the graphics made up for that). However I became disappointed in how GT5 turned out in several aspects, and made it to maybe 23 in A-spec and...dunno in B-spec, and just grew bored. I think it was because it felt like I was progressing too slow compared to previous games, and it felt stretched. (I also got Fallout 3 which held my interest for a long while)
When I first played Forza 3, I was surprised at the better (IMO, I'm more into American cars) selection of vehicles compared to GT5, as well as the visual customization aspects and paint shop. I also like how they set up the events, and I guess I like the physics a little better (not saying they are better of course, that I don't know about) since it sort of "feels" like the cars have more mass. Oh yeah, the sounds are amazing, especially the Grand Cherokee SRT8 (that thing is a Jeep?

)!
This is a bit long, I know haha. But in any case I think they both have their pros and cons, but GT5 just feels half-baked to be for many of the features whereas Forza feels how a racing game should be set up. One day I may pick up GT5 again just to race, and I guess I'll see what the deal is with GT6 before I buy. And that's my two cents!