I have not been able to get my personal BGM to work, so today I did some testing with it. Basically, whenever I have personal BGM activated the game hangs for a full second after starting the race, and proceeds to not play anything at all, or revert back to the ingame BGM.
I tried selecting my own music with any combination of shuffle/repeat settings, made no difference. Different file groupings and selections (ie. selecting only 'rock' music tracks, or a specific artist) had no effect either. Deleting the game's own race BGM playlist only resulted in eternal silence during racing.
After I had given up (but left my settings in place), something happened. I was doing a race, and had screwed up bigtime for a few times, and restarted a few times. At first, this did nothing for the music. Until the 3rd restart. Suddenly I had my own music, working! Sadly this did not transfer to the next event. It is consistent too: every time I start a race the music will not work, until I do 3 restarts.
The stange thing is, when adjusting the volume setting, the game plays music to let you hear how loud it is; the music plays instantly here!
The volume is yet another issue though. Whenever my own music plays either in race or menu, its volume is much lower, even when the user bgm volume is maxed.
In summary, there is nothing wrong with my music or the PS3's ability to play it, but somehow there is an issue with race playback and general volume of external music.