This works the other way around for me.
Laps 1 to 3 are the fastest ones, then I'm losing concentration trying to be faster than the ghost and it takes at least 15 to 20 laps to come back to fast times.
I'm switching off the ghost but the split time display, red or blue depending on whether you're faster or slower than the ghost, is enough for me to brake too late or miss an apex.
I'm usually re-starting the race from the menu (to clear the ghost time) after lap 3 or 4.