That doesn't actually answer his question.
The answer to that question is, of course, that the GT5 replay file is a data file, not a video file. It contains no video or images, just instructions that GT5 can use to recreate any given replay. Youtube cannot read this - only GT5 can.
To record a replay as video, you need to run the signal from the PS3 through a video recording device. That will then give you a suitable video file that you can view on your computer as a video and that Youtube will be able to use.