Been playing since GT1, through all the Prologues, Concepts and I've even dabbled a bit in GTPSP.
GT1 and GT2 were good for their day. Relatively realistic for a console racer of the time. GT3 was a huge step up in all ways, but the GT4 Prologues hinted at something special on the way... and GT4 delivered... GT4 blew GT3 away in terms of realism. Compared to GT4, GT3 feels unpolished, arcade-ish.
The big issue with GT4 is the poor modelling of lift-off oversteer characteristics, and poor ABS/brake-bias modelling... the former made GT3 more fun to drive... despite lacking in terms of realism to GT4. I still loved GT4... because the detailed model gave cars flavor... I could spend hours lapping and finding subtle differences between different year models of the same car, or discover the handlings secrets of relatively garbage vehicles. I had several different GT4 saves simply because the garage size limit meant that I couldn't have all the in-game cars in one save.
The lack in the brake department was fixed with GT5Prologue, which also addressed some of the lift-off issues. And it just got better and better as they tweaked it. Then they took a quantum leap with GT5...
GT5 is the best GT to date in terms of physics. It models differential behavior well, it allows handbrake turns, scandinavian flicks, trail-brake spins, and tire-squealing four-wheel uncontrollable understeer when you overheat your tires. Some wheel users even suggest that there's clutch-slip modelling.
It's not perfect. It still lacks tire deformation modelling, something that NFS:Shift actually hints at, so it's not impossible on a console... and tire-size and profile changes. Plus the silly, made-up units used for downforce, dampers and anti-roll bars makes tuning a whole lot of guesswork... but GT5 is a well-polished product... at least on the physics side.
On the racing side? Well... we can only hope that they find some way to make Seasonals a permanent part of the A-Spec offline experience (they're some of the best races in any Turismo)... and allow us to toggle on tire wear for lower A-Spec races.