100 or 180 macro lens?

I am looking to buy a macro lens but having difficulty deciding between the 100 and the 180. I don't have a particular interest in insects, frogs, butterflies, etc., which seem to be the most common subjects for the 180. My use would be mostly in isolating patterns in flowers/foliage, textures in natural or fabricated surfaces, that sort of thing. For some of this I do ok with the 24-105L at the long end, but it doesn't focus close enough for much of what I want to do.

I am considering the 180 because of the increased working distance, as I am often in public gardens/spaces where physically putting the lens against the subject is not feasible. Somebody also suggested the 300/4 IS for this type of thing. I've never used the 300 (but am familiar with the 100-400) and never even seen the 180. Can't go to a shop and try the 180 where I am, so I thought I'd ask for your views.

Thanks for any help. (Budget is not an issue.)

--
Suvo Mitra
http://suvomitra.smugmug.com