My ideal driver's car is quite a specific thing. It has to be a car that's designed for driving pleasure. Outright speed is irrelevant to a true driver's car, in my opinion. It merely has to be all about the driver's enjoyment of the drive. For me, that means a certain degree of comfort when needed, but also a very, very communicative chassis. The steering must be lucid and full of feel, the suspension should be stable but not harsh, and the handling should be fairly neutral. The engine and gearbox are also important factors in my hypothetical driver's car. The gearbox should be a proper manual, with a precise but mechanical feel to it. The clutch should be straightforward and the pedals should be perfectly set up for heel-toeing. The engine doesn't have to be of any specific type, rather it should be full of character. A 1.5 litre VTEC would be as ideal as an LS7, as long as it is willing to rev and makes the driver a part of the experience. I've not driven a turbocharged car yet, so I can't comment on aspiration, but I certainly feel that older, more analog turbo engines have a great character of their own. In essence, I feel that a true driver's car is one that makes me feel special when I drive it, and one that as I drive it with passion feels like it wants to be driven harder and harder. If I find myself driving down a B road flat out without meaning to, it's a proper driver's car.
There are, of course, things that will detract from driver's cars for me. Width: the smaller the car, the easier it is to place on the road, and the more confident one is when driving it hard. Visibility: if you can't see out of it well, you can't drive it well. Weight: weight saps fun away by requiring everything to be more insulated. More steering assist, stiffer springs without any gains in grip, stiffer dampers, slower acceleration, etc. Lack of comfort: it's hard to drive a car flat out if you're uncomfortable. This is very personal, but I actually feel that a fairly luxurious car is advantageous in this regard. Good seats are a must, but heating and ventilation are icing on the cake. Aircon is a must unless it's a convertible, and even then it's helpful on hot summer days. A good stereo is also on my list, because everyone likes to listen to their favourite driving song while bombing down their favourite road.