David Hobby (The Strobist) once said that he used protective filters on his lenses as a mean of lens caps. And I agree. I simply hate lens caps. What I do now is, I store my lenses with the filters and caps on, but not any filter, but good quality filters, like B&W or a Breakthrough Photography ultra slim filter I bought for my 12-40mm, and when I'm out taking pictures, I remove the caps, and leave them on the bag, and I simply don't care about protecting the front element anymore. If I get dirt or moisture on it, I can rub it with a piece of cloth, or even my shirt, and I'm good to go.
Filters are particularly useful in misty seaside conditions, where you don't want all that salty moisture in your front element. I live in Panama, so we have both oceans at each side of the country, so is sea salt and humidity galore.
I haven't seen any significant detriment in image quality, but I do take them away sometimes at night, when there are artificial lights in the scene, since it can create ghost reflections in the image. Although I shot the night arrival of the first Neo Panamax container vessel to our terminal, months ago, and I left the filters on, since it was raining continuously. I thanked Olympus for their weather sealing that night.
You might as well carry some wipes and do it right 
