I get the same, and it's become really bad since the recent DLC's. I'm on NAT 2, I have a decent connection, and I'm the only one normally using our internet. I get good connection on all other games - such as Battlefield, yet on GT5 I'm lucky to be able to even connect to 3 servers per refresh. Even then, I'll get disconnected by the first minute I'm in the room.
I've tried clearing my cache on GT5 & general PS3. Turning my router off and on again, nothing seems to work for me.