The servers are trash at the moment so it might not be you. I have fibre optic running straight to my home and live just down the road the city's main cabinet. Some nights on GT6 I have real problems, others are fine, but on my PC or other PS3 games, everything is sweet as can be.
The majority of people I play with regularly are having the very same problems too.