One morning, after I took my daughter to school, I stopped at our new soccer stadium to shoot some picture, mostly because the light was nice that morning. Security guy came out and told me photography of the stadium was prohibited and if I didn't stop he would call the police. I said, "Ok," and went home.
When I got there, I emailed the stadium's contact link on their website and asked why it was prohibited to take photos of a public soccer stadium that is visible from the interstate and every major road nearby.
The Vice President of Operations replied that whoever told me that was wrong, and that as long as I wasn't trespassing within the actual stadium, it was perfectly fine to take as many photos of the stadium as I wanted to. She said she would "retrain" her staff on this, and offered me free tickets to the next soccer game. We have a pretty good team: Louisville FC! Here is our new stadium.

Lynn Family Stadium, home of Louisville FC
The Lynn Family Stadium is not public property. It is owned by Soccer Holdings, LLC.
The general rule, is that Soccer Holdings, LLC may prohibit those on their property from taking photos. However, they can't prevent those that are off their property from taking photos of the stadium.
If you were standing on stadium property (for instance their parking lot), then the security guard may not have been wrong when he told you that you could not take photos on stadium property.
The VP of Operations was also correct in that if you were not on their property, then you are free to take photos.
It's not clear from your post where you were standing. If you were standing on a city/county owned street/sidewalk, then you were likely within your rights. If you were in their parking lot, or on a road/sidewalk owned by Soccer Holdings, LLC, then they have a right to prohibit photos and to ask you to leave.
Of course, your use of any photos may be limited if the photos contain trademarks, copyrighted elements, or recognizable people. Being allowed to take a photo is not the same as being allowed to use the photo however your like.
A similar situation applies to private homes. I can take a photo of your home from a public street, but you can prohibit me from taking a photo from your driveway.