What you want to look at is the Track Matte Keying. With a Track Matte, you can apply other Effects, like Blur to the edges.
If you have Photoshop (the full, or Elements), you can create the Track Matte and "distress" the edges. This can be done by the application of various "edge effects" (sold separately from companies, such as PhotoFX) or by hand. For instance, you can create the look of a Polaroid "transfer print," or similar.
Read up on the use of Track Matte, and then begin experimenting with altering the hard edge. You also might get some pointers from:
Both show other uses for Track Matte Keying, but the principles can be applied to your use. You will essentially be doing a PiP to create your montage, but will then alter the "window" in the Track Matte.