I just realized I blew off replying to your post. Sorry about that.
Disclaimer: I'm NOT a lawyer. This is just an area of interest to me...
There's always the chance things can get complicated. The bottom line is, in a public place like a park--even near a swimming pool or kids playing on swings--you have a legal right to shoot photos. There is no expectation of privacy in such a public space.
The flip side is that there's no practical way to enforce your own rights if the police decide they want to give you a hard time. Assume you had kept shooting when those guys told you to stop. If they touched you or grabbed your camera away, you could get them on an assault charge, but they'd probably just call a cop to back them up. If the cop decided he/she didn't like your look, he might tell you to leave, and if you didn't, you could end up with a disorderly conduct charge. In the long run, you'd probably get out of it, but you'd have to go to court and maybe end up with some legal bills. Most of us, in those circumstances, will probably just walk away.
One thing you can try is carrying this document:
http://www.krages.com/phoright.htm
It's written by a lawyer who's also a photographer. It outlines, in general, your rights.
I do find it ironic that, in an era in which we have police cameras on many streetcorners, photographers are being accosted for acting in a perfectly legal manner.
--
Peter
nifty50.net