I'd imagine it's because of the above reasons, and a few others.
1) We no longer see fire-breathing monsters in GT category racing outside of the occasional rather poor Trans Am style series. Professional and successful GT sports car series are mostly a somewhat "tame" racing car by comparison.
2) Because of the above, in a mixed class series, the GT sports car is rarely the top of the pecking order, and thus while a driver can compete for a class win, you're never going to have the outright overall victory which is something every competitive driver would want. A GTE Pro/Am victory at LeMans is great...but it's not as great as an LMP1 overall victory. This is an issue you don't run into in a variety of other motorsports which are not mixed class.
3) I don't think a single racing driver would be disappointed in driving a factory Porsche GTLM/GTE car...but how many drives like that exist? Less of those probably than the Indy Car field, etc.
I think most competent drivers could and do thoroughly enjoy driving sports cars or GT cars, but everyone would like a few shots at a world championship, or the Indy 500, etc. It gives them bargaining chips in their career going forward.
The pay is an issue, but that's been the case all across motorsport with very few exceptions. The days of egregious paychecks from big tobacco and alcohol companies are long gone. Teams are struggling to find any sponsors, let alone big budget sponsors. That's not saying you can't make a livable income racing, but the "big money" days of the late 80's and early 90's are gone. Listen to any of the podcasts and interviews with even Indy Car guys from the 80's/90's and they were making 5-10x what the average Indy Car driver makes nowdays. It's a changed market.
Needless to say, unless we see some kind of GT1 style crazy resurgence, no one in sports cars is making a crazy salary.