It's good to turn it off if it's 0.1s ahead of you and you're trying to get a good run through the last few corners to get a good start on your next lap.
As mentioned already the default map is "Up" on the D-Pad... but the Ghost Car function is mappable in Options.
The trouble is that there are a lot of mappable functions and only so many buttons...
So you CAN turn it off... this is very helpful as I find they more hinder then help my efforts on license tests... they throw me off for a split second and then I can't see where I'm going properly!