I think it's lag. The game might not "see" the same things we do, or that other players do. I've lost DR and SR while gaining a bunch of positions and running a clean race. Other times I've been run off the track and had contact with other cars, yet received the clean race bonus and a bump in ratings.
I've also been flamed after a race for being dirty when I never ever touched anyone.
The game sees exactly the same as you do. Each console calculates its own collisions and penalties and only sends your car telemetry to all other players. The server literally does nothing.
Lag can be a big factor in how different players experience different things. If you or the other person has lag, their car telemetry arrives late on your console, which means your game extrapolates old data to figure out where the other car could be right now (before confirmation is received of where it actually is). So you can hit a car that's not really there and the other person will never have experienced the collision. Sometimes your console tells you the other person got a penalty yet at the end it turns out they are penalty free as each console can experience the collision differently with lag. This also happens the other way around hence getting called dirty while you have no clue why. (It could of course also be mistaken identity)
The penalty system is also very sensitive. You might not experience a collision, yet a touch of bumpers is enough to get an SR deduction. So you can lose SR at the end without ever feeling a hit.
DR gains and losses entirely depend on what the DR level is of the people you finish in front of and behind. People finishing in front of you take 80 + ((your DR - their DR) / 500) from you, you get 80 - ((your DR - their DR) / 500) from people you finish in front of. Basically if you beat higher level players you gain more points, yet if lower level players beat you you lose more points.
If a player punts you off the track, you lose SR, yet if you don't get SR Down or a penalty the game believes you were not at fault and you keep the clean race bonus, plus you also receive a free pass go off track temporarily. (Unfortunately this is abused by some people, dive bump hit in the right spot and take a free shortcut over the second part of a chicane)
It would be better if the server was involved with penalty assignment as you can't trust penalty info for other players at all, neither the flashing sign, not even the red dot at the end of the race.