Give up photographer's rights or risk being sued?

Although I'm not using this forum to seek legal advice, I'm looking for some insights that may help me maneuver into this somewhat stick situation with a circus company.

A group of photographers and myself are involved in a community project to showcase the everyday life of our city. When a circus came into to town, I thought it would be an unique opportunity to photograph the event and post them on our community blog. The agreement was to grant us access to photograph the event in exchange for providing them copies for private use only.

A week later after the event, a company representative e-mailed me for a copy of the images. He said they intended to use them for their company website, which basically the hub of their e-commerce to do ticket sales. I reminded him via e-mail that the images were to be used for private use only. He and the company keep insisting their e-commerce website is classified as private. They keep insisting that they "don't normally allow people to photograph the event." However, we saw no sign prohibiting photography nor did any of their company officials prohibit other audience members from snapping away with their cameras.

So far, we have disagreed on the definition of it and I keep referring to the previous e-mails where I explicitly stated that the copies of the files were for private use only to no avail.

Now they are threatening to turn this matter to their legal department. I feel they are trying to intimidate us into turning over the images. However, we're not budging on the issue yet unless this keeps dragging on. We are even considering billing them for the commercial usage of the photos to get the point across.

What is everyone's opinion on this? We have not even published the images we took at the event, nor do we intend to.