next lens for my 5D Mark ii

Wondering on what my next lens purchase should be. Right now I have a 50mm 1.8 after getting rid of my 85mm 1.8. Though they were to close in size. Do you think think that I should go with the 100m f2.0 or 70-400 f4.0 non IS. Most of my shots will be of my kids, travels, and an occasional shots of people, portraits etc.