well this applies sometimes but whichever side you go off on, when you get back on, stay on that side, it could cause confusion. Some argue to "always pass on the left!" but if someone goes off left (brands hatch turn 1), and they come back on, you don't know if they're going to cut all the way right to "get out of the way", so the person who went off should stay on the edge while getting back up speed and you (passing) see them coming on so you should change your line just a bit to pass on the right, then resume the normal racing line after.
in summary, yes I agree with you and I've rear ended many people who have gone off (didn't ghost) and come right back on but cut out of the way while I try to pass but they end up cutting me off so I give them a lovely speed boost. sigh, such is life.