From the perspective of someone who has both games:
GT5 has the 'better' graphics, FM4 has the 'prettier' ones. GT5 is more realistic looking, the tracks look more like their real life counterparts (especially the 'ring), the premium cars are more detailed (even autovista comes a hair short of GT5's premium cars). GT5 does score a minus point with the standard cars which should not have been in the game at all in this state, imho, it makes the game look very inconsistent at times.
The UI is much more streamlined in FM4 and much more responsive. Clear win there. While the GT5 UI has its charms, especially for non-fans of the series it's simply not good enough and even for fans (i got the signature edition) it just is a bit... odd. I vastly preferred GT4's "charming" menu style, or even GT2's.
The driving feel is a toss-up between both. GT5 has the slightly "weightier" feel, especially when using the controller as it has rumble, this is aided by the more pronounced weight transfer visually. FM4 has the better tyre feel, despite not having rumble, the audio cues are far better. However, both games are very nice to play, the differences are just differences, not in favor of one or the other. Overall i would have to put GT5 just a tad ahead in physics though, as far as i can tell, but this is a detail that only the hardcore players would notice.
The game structure is again a clear win for FM4. While still not perfect it's miles ahead of GT5's incredibly grindy structure. The special events however in GT5 are well done for the most part, and it does offer some extra little things that FM4 seems to lack (licenses), aswell as off-road racing events and the track editor. They do contribute to the game's (GT5's) schizophrenic nature though, FM4 feels like the more polished *game*.
The car lists are nearly equal. GT5 has a massive roster, though there's a heavy bias towards Japanese cars and only 200 or so are premium cars. FM4 has less total cars but they're more consistent, all featuring cockpits and it's a selection from all over the world with less 'duplicate' cars (40 variations of the R34 is just too much). FM4 has the slight edge here because of this.
The tracklist is a fighting point for both, though FM4 has a heavy emphasis on US tracks. GT5's fictional tracks are somewhat better designed, the tracks themselves resemble their RL counterparts better aswell. Both games have about the same amount of variation. FM4's tracks have nicer scenery, but especially the 'Ring just doesn't feel right since it doesn't really resemble it's RL counterpart. In terms of track surfaces, GT5 has the ever so slight edge by modelling bumps etc a little better. I'd give this to GT5, if ever so slightly.
Online is a clean sweep for FM4, no question, same with customization.
AI is a little iffy for both as far as i can see.
GT5 is a schizophrenic but obsessed and passionate driving game.
FM4 is a smooth and polished racing game.
At this point... I can honestly not say i prefer one over the other, i'd rather have both. Both have their great moments, both have their little nags, but together they're pretty damn good.