I think it does move it for every car, but some cars mirrors are just in crap places for visibility. Personally, I'd rather have a closer FOV so it feels more natural, than having a far away FOV so that I can see the mirrors. I just use the look behind button when I need to look back.
That's the same problem I have with the FOV in PCars2. Once you adjust it for one car it totally messes up another one. I adjusted one car in PCars2 then I went straight to one of the Astons and it looked like I was driving it from the back seat. So I haven't touched it since.
It's too bad that that adjustment can't be moved into each individual car's setups in both games.