How to carry OM-D E-M5 to be accessible when backpacking?

I used a chest pack similar to the lowepro attached to the d-rings on the backpack straps for quite a while. What I didn't like was that it flopped around a lot and that I had to disconnect it every time that I wanted to take the pack off. I guess it also depends on your lenses. I always had the Panasonic 100-300mm on the G2 or Em5, or the 300mm f4 on the Nikon D7000, so I needed relatively large packs.