unexpected viewfinder issue

Has anyone else noticed that bringing a camera to your face to look through the viewfinder draws attention to the fact you are taking a picture in a way it didn't in the past?

I bought the Oled finder for my NEX-5n and other than the crazy expense I thought I was really going to like it but I discovered as soon as i brought the camera to my face in a group of people that people would react very differently than when I use the camera from waist level looking down at the screen. Am I imagining this?