Wow thank so much for your quick response, I really appreciate it!
But sadly the techniques you showed me above do not give me the wished results.
To make things more clear I made an example to show the problem (please note that I took random pictures I made a while back..):
As you notice on the first picture you got the sharp picture in the back, and the blury one above. (I actually tried your tipp here with the smart object). But as a result you get a messy transition vom sharp to blury.
On the second picture I used the Tilt-Shift filter to show the wanted result.
But now I want to create a more complex transition, not in only one direction as shown with the tilt-shift. For that I'd like to create a black and white mask as mentioned above, so a blur filter can decide on how much it makes each individual part of the picture blury, dependent on the grey scale of the mask. So deeper grey gets more and more blurry..
The "messy transition" is because a mask only gives you an overlay of blurred and non-blurred, with gradual transition. So you get ghosting (for lack of a better term).
It seems you may want to look at a dedicated plugin for this. Alien Skin Exposure has a "Bokeh"-function that will do this.
Before you look outside Photoshop, try these
1. Lens Blur Filter
You can paint black white and shades of grey on a mask then use the mask as a depth map for the lens blur filter
2. Use pins and field blur
Alternatively - you do though. take a look at Field Blur using multiple pins, and see if it will give you the output you need . It varies blur in a smooth way and, although not using the painting method you were asking about, it does let you control the blur at different points that blend into each other - just keep adding pins to build up that variable blur.
Quick and dirty example below
Oh. my. god. Thank you so much, I was never aware of the Lens Blur Filter, but that is exactly what I was looking for!
Thank you for your time and effort!
Two other points though. First, remember to disable the mask when you are done - so that it doesn't hide the layer. Second, work on a copy of the layer. Lens blur is destructive and, unfortunately, won't work as a smart filter.