I seriously don't think the PS3 is anywhere near being "maxed out".
Developers just haven't unlocked its full potencial because:
- when designing a multi-platform game, developers use the X-Box 360 as the lowest common denominator, since it's technically inferior and easier to program for. So the PS3 gets held back on multi-platform games;
- regarding PS3 exclusives, I don't think developers have fully mastered designing for the PS3 yet, because of it being harder or just plain different. No doubt PD and, for example, Naughty Dog, have taken hardware use to a whole new level, but I still think there's plenty of uncharted territory to discover. (pun intended

)
That being said, though, I still think PD will choose to prolong GT5's life through DLC, rather than develop a new game for the PS3.
I suspect there's just so much content they want to put in GT6 that they won't have the time to do it in the PS3's lifespan.