I have both games and personally, I think Zero is better than 3. One reason, the AI in Zero is better. I find the AI in TXR3 to be pretty brain-dead in comparison sadly. It doesn't have the characteristics of Zero's AI, where racers have distinct driving styles. Some racers in Zero will block you to prevent you from getting in front of them and will try to get around you if block them, making driving skill pretty important. In 3, if you block someone, they'll just ram you.
I also think Zero allows you to do more to your car, performance wise. You can't have a 1,000+ HP Viper or Skyline in TXR3 for example. But I think the reason you could do more, is because none of the cars in the game are licensed.
TXR3 does have engine swaps, but you need to put 2,000 miles on a car to do them, because the game will be ridiculously easy if you do it on the right car.
But I'm not saying TXR3 is a bad game, it's really good. It definitely has its own perks that make it worth playing. It adds more cars outside of Japan like GM, Ford, Lotus, Mercedes, etc. and you have three cities to dominate instead of just Tokyo, to name a few things.
About the physics, yeah, the game is definitely not going for any kind of realism. I have a 1,000+ HP Porsche and it's easy to control.
By the way, if you get stuck, feel free to message me. I'm thinking about playing through the game again myself.