'In digital photography you have what is called a 'dynamic range'...it is a measurement of light that any given Camera can capture...this is displayed on a scale known as a 'histogram'. Basically the reason why you get this problem is as you pointed out; a lack of smooth transition in tonal range, because the distribution of light tones is heavily favoured to the right with the scale looking like--->(Dark--->Light):
http://d2f29brjr0xbt3.cloudfront.net/525_lightexposure/04.jpg
To help reduce the effects of this, pro's shoot in RAW format and use a technique called 'exposing to the right (ETTR)'. This is a clever way to maximise your Camera's image sensing capabilities and allow you to have greater tonal gradation in shadows.
Because there is a limited tonal range in shadows, the transition between tones can be as you said 'abrupt'. It doesn't look very good....now this is where HDRI comes in...