Could you post an example of what you would like to do? That can be either very simple or very difficult.
The human eye is very sensitive to strong transitions so the solution is often to blur the selection before applying the effect.
For example, imagine that you want to blur an object in the background.
(1) make a copy of the layer (never work on the original!)
(2) make a rough selection around the object using the free selection tool. you do not have to be very precise.
(3) blur the selection (not the image). The tool is called 'feather' in Gimp and probably in PS too. Do not hesitate to give a large value (e.g. 20, 40 pixels or even more if you can)
(4) if the background object if very close to a subject which has to remain sharp, then I usually switch to 'selection mask' mode and I carefully erase the parts of the blured selection that overlap with the subject.
(5) Finally, apply the required effect (blur, unsharp mask, ...) in the selected area.
--
Stephane - Minolta 7D - Sigma 28-105/2.8 - Sigma 105/2.8 - KM 100-300 APO
http://www.chauveau-central.net/stephane/gallery/