If he hit you while he was braking but couldn't stop in time, he most likely wont get a penalty (in GT5). You got the penalty because you hit the wall. What I've learned from doing time trials is that if you try to avoid hitting something but indeed hit it, they will wave a penalty. If you hit something directly without applying brakes or trying to stop (carelessness in most cases but not yours), you will get a penalty. There are flaws in the system though, however, most of them can be avoided if you try to avoid (even though it might be impossible).
There are a lot more penalties than this, I'm just explaining your example.