As an avid cyclist of ~30 years, there are a number of reasons I steer clear of the bike lane proper (which is to say I ride on the inside dge of the bike lane or the outside edge of the traffic lane).
Small debris. Regular traffic flow naturally sweeps offending materials to the bike lane or shoulder, and riding over these materials is unpleasant at best or downright hazardous at worst.
Large debris. It isn't that uncommon to come across larger materials (tree limbs, construction waste, general refuse) that one must swerve to avoid or risk serious injury, and if one is riding in the center or outside edge of the bike lane, the only course of action may be to swerve into traffic, while when riding at the inside edge near the traffic lane or outside the bike lane, avoidance can often be accomplished by swerving away from traffic.
Road surface. Bike lanes and shoulders tend to be treated to a thinner hot mix asphalt layer that weathers at a higher rate and results in an exposed aggregate layer that's not unlike small debris, except that it's a constant condition not subject to further sweeping either by traffic flow or the elements.
Refuse containers. On days that refuse is picked up, the containers holding it are generally left on the curb, but residents often push these cans farther away from the curb, seemingly unaware that the collection trucks are capable of getting closer to them if necessary.
Parked cars and opening doors. As long as I've been riding, I've had the misfortune of having doors open up on me on numerous occasions, two of those resulting in collisions and injury. Too often, those occupying the side of a car nearest traffic use the bike lane as something for their convenience, oblivious to its true purpose, and open their doors without ensuring the coast is clear because the bike lane provides a buffer from the traffic lane.
Motorists. This may seem counterintuitive, but there seems to be a significant degree of complacency among motorists with regards to the presence of dedicated bike lanes, and they are often oblivious to cyclists using these dedicated lanes, assuming that since the lane is there, there's no reason for a cyclist to occupy any portion of the traffic lane. All of the above are reasons for exactly that, though, and a cyclist that is forced to react to any of the above can easily fall victim to oblivious motorists.
In my role as a motorist in the presence of cyclists, I'd much rather know they're there on the edge and exercise caution accordingly than have them be forced to jump out in front of me unexpectedly.