I've been reading a few articles lately that I think relate back to GT5 in one way or another. In one, the author was arguing that writing completely new software from scratch is a terrible idea (his case study was on Netscape Navigator) because you put yourself in a situation where you're not supporting your old product (not a problem with GT4, though), you can't launch your new product until it's finished (obviously) and all those years of bugfixes and corrections are lost because of unintentional obfuscation of code.
This made me think that maybe PD started writing new AI, physics and graphics engines from scratch, realised that it would take several years of testing and rebuilding to get it into a state that they can release, then decided they'd have to scrap it to get a game out in 2010. It would explain why GT5 is as thin on content as it is for a 6-year development and why it's not that different to GT4.
Then I was reading about Duke Nukem Forever, or specifically why it took so long to release. It turns out the project leader when development started kept seeing new games and wanted to add as much as he could to DNF. After a while the technology they were using became obsolete, so they had to start all over again on a different engine. This kept going until the people giving him money got fed up with it and hired someone else.
KY seems to me to be the same kind of person, but it's understandable. He clearly knows racing, cars and the enthusiasts that play GT5, so it would hardly be surprising if he wanted to add as much as he could; problem is that when it comes to racing, that's a whole load of stuff. He even said he wanted another two years on GT5, and he would have added FIA GT cars for a start. Sony probably got fed up with that approach and told him in no uncertain terms that he had to release a game in 2010, so he did. I do realise he's pretty high up in Sony itself, but shareholders tend to have a lot of pull, and maybe he even realised himself that PD couldn't afford to put it off any longer.
Then there's also the fact that there was a trailer from a while back that showed a flag system, does anyone remember that? I vividly remember seeing a small animated flag icon, but I can't remember what else was in the video, so I don't know for sure that I wasn't imagining it. Still, GT5 has already shown itself to be quite a modular game, it's a sure bet that there is code in there somewhere for a flag system, but it wasn't finished in time for GT5's release.
Now, I'm not trying to excuse PD for GT5, I think it should and could have been far more than it is, but when you read about things like that you do wonder exactly what happened with GT5's development. I mean, it's highly unlikely they were sitting around for 4 years after GT4, then woke up one day to start work on GT5, yet GT5 only seems like 2-3 years of development time.
Before you say 'that's all well and good, but how hard is it to implement x feature or y car?', I would like to remind you of the drought of events; how hard is it to make up new events if they can release a 'new' seasonal 'every' week, even if they are A-Spec events with different circuits and rewards?
Maybe by November 2012 we'll have the game KY always intended to release via these incremental updates, or maybe we'll have something halfway there and GT6 will be out sooner than 2016.
Apologies for that huge post. TL;DR- There's plenty of reasons why GT5 fell short of a sim, PD's lack of understanding of how sims and/or motor racing work isn't necessarily the biggest reason.