Oh, I agree. There are some game types that get away with rehashing gameplay better than others as well. For example, story driven games need only be competent in terms of their gameplay and mechanics, because the real meat is elsewhere. A game can be completely ordinary to play, but still fun with a great story.
Unfortunately, for racing games traditionally they're almost entirely focussed on gameplay to the exclusion of anything else, so the fun has to come from gameplay. In those sorts of games you either innovate, which is the "accidentalness", or you're just another game.
I guess good developers build a solid base, then screen tons of new ideas for the ones that are really the most fun.
For me, story needs to correlate with the gameplay - you cannot just paint different stories over the same gameplay and expect it to gel in the same way. It's a matter of the cohesive whole, and requires serious design talent to achieve without masses of iteration (which still occurs with said talent). And most stories in videogames that I play are throwaway anyway, so the gameplay still counts. The only ones whose story matters most are those so-called "not-games", where gameplay is deliberately limited. Then you start getting into a whole discussion about "what is gameplay?", and for me the easiest answer is "all of it" - at least it should be.
When I say "accidental", I don't mean blindly stumbling around and going "ooh, this is good" when you find something that works. I mean, people know roughly what they want to achieve, but it's impossible to get it right first time, and to predict interactions with other systems (that holistic view again.) By taking mechanically similar games and applying different stories, or whatever, you are neglecting those interactions at the first step. That's what immersion is, that sense of cohesion in all of the game's systems.
One thing to be wary of when "screening for fun" is that a lot of developers use focus groups and other things that are popular in the film industry and elsewhere. This technique doesn't really work there, and is also responsible for a lot of the cabbage water that comes out of the games industry, in my opinion. "The most fun" depends on who you ask, so like you go on to say below, it's better that people / games are allowed to be what they ought to be, even if that makes them a total failure (usually a marketing issue, like the recent Thief game - not brilliant, not especially Thief-y, awful story and poor correlation between that story and the gameplay / level design etc., but better than the marketing implied).
The lines are blurring much more than they used to. Pre-3D graphics there were pretty strong limits on what was graphically possible and what sorts of games worked within those limitations. Nowadays, if you can imagine it you can probably have a fair go at doing it, or so it seems. A lot of great games are the results of taking what worked from two "genres" and sticking them together. And most of the best games largely ignore what people might expect them to be, and just do what they want.
GT arguably did this originally with aspects of racing games and RPGs. Unfortunately, the aspects that were so popular then really aren't so much now. Gaming has moved on from what was popular in 8-bit RPGs, but GT is still at heart the same formula. Possibly that's why some people feel that it's dated and needs an overhaul.
I still think Kaz is a perfect guy to have at the top. He's passionate, and from people that have met him extremely charismatic. But that doesn't mean that he needs to retain full creative control. He can be great at his job by promoting GT, supporting/motivating his staff and providing overall vision for what GT should be. Then people closer to the ground can figure out how to best make a game that supports that vision, and whether it's actually possible given the constraints that they know they have.
I remember an interview a long, long time ago where Kaz was talking about the importance of 1080p AND 60fps in GT5. He was mentioning how his engineers were saying that they could do one or the other, but that doing both just wasn't really going to be feasible. And Kaz said that he stuck to his guns, and eventually they found a way. The tone of the article was that it was his drive that pushed his team to make that breakthrough, but it struck me at the time as an example of not really listening to what the people around him were saying. Yeah, in the end they sort of pulled off 1080p/60fps, but it's pretty shaky and it remains so to this day.
I'm sure he has great staff around him, but perhaps he needs to pay a bit more attention to their opinions too. The days of big games being the creative vision of a single man (or woman) are largely over. One person simply cannot be expected to think of everything in a game of this size, and nor should they try to.
I think in more recent times Kaz has learned to do just that. Obviously he appears to take a personal interest in every aspect, and I think that's better than being only a "director", you definitely need that underlying technical knowledge (which would imply he knew the challenge with the resolution and framerate), the trouble is people like to take the easy route too often. The recent physics improvements and partnerships with the associated companies are probable promising evidence of Kaz trusting others, sound absolutely has to be, too (once it gets here). I can imagine people approaching Kaz, and him saying "what's the best way you could do it?" "What's stopping you now, what about the future?" Start now, and the future will be here sooner...
The developers at id constantly complained about Carmack's insistence on 60 fps in Rage (it uses a nice resolution scaling feature to manage the per-frame burden from moment to moment to maintain that figure better), and I'd say that's one of the best things about the experience, and will no doubt influence future games (or maybe they'll just try aping the fluid animations, and will be left with "floaty" nonsense in 30fps, and give up, because they missed the point). It's a nice thing to have, and graphics can be turned down a bit to compromise, it's not a real issue when that extra responsiveness counts for so much in an action game. Racing games benefit for similar reasons.
If we were just content with what came before, there wouldn't be any progress. It's better that these people work hard on solving such complex issues of balance, because that experience can be put to better use on future issues. It's better for them, usually, too. I'm sure there are plenty of examples outside of graphics, and I expect the new sound generation method was not without its detractors and difficulties, but arguably the game needs it more than better graphics, and if it's half as potent as I expect it could be, it'll be a game-changer. Whether people will think it was worth it, I couldn't say, even if I knew exactly what to expect.
I know that a lot of people think GT should be about racing, purely, but I don't think it ever really was. The racing was a backdrop for the relationship the player had with their car, specifically the driving. Lose a race? Was it me, or was it the car? Do I need to find a different car? And that is more in the sense of the aforementioned RPGs, (which I never really liked), more than something like GTR (which I also never really liked). It's like knowing you should be able to smash the gold time on the seasonals with an untuned car, but at what point do you admit your inadequacies and give in to making tweaks? Then of course there's the separate game with the leaderboards, which is more "traditional" tuning fare, but it still causes players to use cars they possibly wouldn't have otherwise.
Obviously, I'd prefer a more free-form approach to racing organisation in the game as an additional feature, and a return to the scale of player-car interaction encouraged by the much larger selection of "backdrops" (race events) in previous games, but I expect there is a real reason it's limited in the last couple of games: completion rate mainly, features to be added in the always-near future, the old formula feeling stale etc. Ultimately, there is probably an element of PD not wanting us to have that level of control, because you'll be less likely to try out so many different cars, which again is more the point of the game - the sheer breadth (not depth) of car selection. With new gameplay features (perhaps those planned for the near future), maybe that broadening of palette will be more natural for the general player, and any free-form racing / career aspects might follow.
Done in that way, PD can add features to GT that appeal to people asking for it to change, and simultaneously freshen up the feel of the game with their own ideas, without them having to change their fundamental approach and "philosophy". It'll just take time to find those solutions, I suppose, unfortunately.