I've never experienced any lag when people join my room on GT5. I don't recall it happening on FM3 either.
How does it work if you want to join a friends lobby on Shift? Do you still have to wait then? I have quite a few friends that I race with on GT, and if they weren't able to join whilst I was racing, I'd probably never see them online due to the fact most races I do are 10 lap minimum with only 5 mins between each race for qualifying.
I'd like to pick this game up once the input lag issue is sorted, but this would be a big game breaker for me.