I've not had any major connection issues or crashing in Quickmatch but I have only ever entered one race. In said race I did noticed that the frame-rate was lacking slightly because of all of the ghosting (which happened almost constantly) and there was a bit of lag in the lobby.
But, it is the worst idea ever, well it's a good idea, just poorly executed. It really doesn't show off how good online play can be in GT6, it's just full of complete arseholes who can't drive and just crash into you, and finding a decent player online is rare, especially in a QM scenario. So, it's best to avoid Quick Match in all honesty, especially if it has this many issues going on.
They need to implement a Karma system into the online. Something that can register and follow each time you crash into someone, each time you cut corners and what your average lap times are so they can match you against similar players. So if people want to play a Quick Match and be an arsehole then they get put with other arseholes, but if people want to race properly but don't want to set up a "Clean Or Kick" lobby then they should be able to be matched with similar people and not get smashed into.
Best thing to do if you want to have good online racing is to find a group or league and try them out. That's how you enjoy online without these crashing, lagging (to some degree) and terrible driver issues.
To answer the OP's connection issue. There is a weird bug in GT that sometimes you can't connect to other players depending on region or your internet capabilities. We have a member in my racing group that can rarely connect to everyone even though he's in Scotland. Unfortunately I don't know of a fix for this.