I would say, don't "switch" until you've exhausted the appeal of GT6. Meantime, play BOTH. Both are great games, with much to offer.
That said, I have actually "switched" from Forza 4 (and 3, 2 and 1) to GT6 (and 5) because after a zillion hours of gameplay, Forza had become stale.
If you can get beyond the religious attitude that one game MUST be better than the other, and accept that they are different and that your gameplay in each will go in different directions, you'll have a lot of fun.
They each have their own unique differences.
For example, GT6 has a heap of tracks, weather and time of day which FM4 does not. So that provides a wonderful variety of driving experiences.
FM4 has a livery editor which GT6 does not, and you can burn endless hours in artistic pursuits or just dress up your cars with a few well-chosen shapes and colors.
Both have the ability to photograph cars and tracks, with GT6 having a far faster and more friendly workflow. Both can produce 8.3 megapixel images. Both are good, but different.
FM4 provides an auction house where you can buy and sell cars. Some players gamble there for bargains and resell them to earn quick credits. You can avoid the hard work in the livery editor by buying a pre-decorated car. Elsewhere, you can buy and sell livery designs (or give them away for free), and the same for tunes.
If the speed of evolution and scope of GT6 is enough to provide sufficient "freshness" to keep you interested, don't leave it behind, continue to play both. I know I have barely scratched the surface (pardon the pun) of dirt tracks, and weather/time change as an example. I've driven nearly 14,000 miles and taken nearly 2,000 photos, and I still don't smell any "staleness".