Motorsports will likely always be a niche sport, and one very big reason is that it is a very complex and intricate form of sport. There's so many variables involved, which almost makes it very inaccessible for the casual observer to get into, especially because the average expectation of a motor race is a pass for the lead twice a lap - all race long, fourteen spectacular crashes, thirty-six beautiful and completely unique cars, at least one pit fire, and a brawl after the event is over. Instead, the average race isn't terribly thrilling to someone with no initial interest in automobiles, competition, strategy, and other obscure facets of the event/series.
There's a huge disconnect between the masses and what is perceived about the drivers; they're rich folks doing something that is generally not thought of as an athletic activity - driving a car - which is believed to be something "any" teenager can do once they get their license. Once I explained to someone that something like the Daytona 500 is an experience you can try for yourself: Go take a road trip within Florida in July, but you can't turn the air conditioning on. Put down all the windows, and suit yourself up in long underwear and your best three-piece suit and tie. No bathroom breaks, and you can't get out of the car. No radio, and you have to increase the volume of your engine about four-fold. Plus, we're taking away the power steering. And forty others are doing the same thing, and don't care if they bump you out of the way so they can also finish first.
The early days of racing were, in fact, only composed of royalty and well-heeled enthusiasts, because motor vehicles were scarce and expensive, and therefore only the wealthy could afford to compete...never mind they were basically fearless conquistadors using scarcely-tested machinery over roads that likely hadn't been proven for automotive use! The dangers of motorsports are apparent in other forms of sport, but nothing like the casualty levels in auto racing to this day, which sort of relegates it to a bit of a blood sport in the public's eye.
I think that's probably NASCAR made such a huge push to be that "5th sport", part of which was that the drivers used to be average folk with a smaller budget to get their way tot he top, and many still worked on their farms the Monday after the race, or went hunting, et cetera. Some of them really connected with their fanbase, but as the sport polished itself so much to the way to the top, the personalities dissolved into every other average athlete, more concerned about his sponsor appearances than relating to their fan base. And that's what I don't get; the superstar stick-and-ball athlete isn't any different than the superstar racing driver...why do the fans think there's so much difference between them?
The average person doesn't know too much about the way their own car works, and therefore, doesn't really care to know more about the technicalities of aerodynamics and heat dissipation. On the other hand, does the average football fan really care much about how the turf was grown? The exact type of wood in a baseball bat, and how to manufacture one to exacting standards? Sure, there's always a few fans who want to know everything, but most team sports are just as accessible to the public without knowing those things. Auto racing in its various forms demands a little bit more knowledge because the car plays a role that's just as important as the driver, which unfairly "reduces" his role in the public eye.
So there's just a couple of points. A lot of people just think it's plain boring, which explains away a lot of things.