When binoculars, monoculars, telescopes (sometimes), rifle scopes, spotting scopes, cell phones, and other cameras are described as being "number X" it's not unreasonable to assume that a big telephoto camera lens could be described as the same, especially since extremely few people (us) think of lenses in terms of focal length relative to 35mm film/full frame equivalent. It also doesn't help that the "X" can mean either "power" or "times zoom relative to the base focal length" I really don't think it's a dumb question at all, just one that they don't know they don't understand, and there's no reason to be snotty about it.
I think it's best to have the answer ready, something like "it's not really measured that way when it comes to camera lenses, but if it were compared to binoculars, it'd be about 10.5 power but about 21X the zoom of your cell phone's main camera". I've only had to use that a couple times, but it worked.