Well, I gather Mmiles' problem occurs mostly in downshifting. Maybe too many things to do with your hands at once if using a controller. Depends upon how you map the buttons, if you're downshifting with the right hand and braking with the left trigger, seems like it should be okay, or at least a matter of practice.
Shifting up is easy. The main problem I've encountered when gearing down is occasionally shifting down all the way to 1st instead of 2nd, which depending upon the game, suddenly breaks traction in the rear, violently spinning the car.
My advice is to play with a wheel if you can, it makes changing gears very easy.