Many photographers carry much bigger and heavier camera bodies face down, attached to much more expensive lenses than yours without a second thought. I can't imagine for an instant that a holster bag would result in damage for that reason.
I reckon you could probably stand on the back of your camera without damaging the lens. Your LCD might shatter, but the lens would be fine.
If it helps put your mind at rest, Canon sell their own holster-style bag for DSLRs. I can't imagine they'd do that if the design was fundamentally flawed.