40D Lens Choice - What would you do?

I've decided it's time to upgrade my lenses on my 40D and have basically decided on the following two, which is all that my budget will allow -

EF 17-85mm f4-5.6 IS USM
EF 70-300mm f4-5.6 IS USM

My quandary is: Should I stay with the 17-85mm or instead go for a EF 17-40mm f4 L USM? Afterall, "IS" is probably not needed at these focal lengths, so would I be better off with the "L" quality, but is the 40mm too limiting?

Swapping this lens will still keep me within budget.

I'd be interested in other peoples opinions and comments on this one.

Shane.