Update: It's been a week since I stopped clearing my cache on a regular basis. There may be something to this. A cache should be generally a good thing, unless it has corrupted data.
I've only had one issue since I stopped clearing it and it wasn't a total freeze, it was just a black screen while my car was doing its rolling start. It did clear up after a couple of seconds and required be to restart the lap again (practice mode).
Oddly, it was on a track that I had been re-running all day with a car that I had been driving for about an hour. It should have had all of that info ready to go, but for some reason it didn't. I think there must be a different cause of those black screens.
I also wonder why the cursor is locked during menu changes, or why the damned menus take so damned long when they are just menus. Who knows?
Anyway, I said that I'd be honest about it. I haven't had a total freeze up yet, but I also haven't noticed a really big change in loading time either.