Are you using a hackentosh? Or else how have you managed to get a gtx460 with a Mac?
Sorry if I'm off track, but I didn't think a Mac could use a GTX460.
If you have performed some modification to the system or the gpu card, it could be part of your issue?
I do have a hacked GPU and it could be a factor but I don't think so since everything is performing flawlessly and enjoying the GPU Boost! I read on the forum that there's some incompatibility between GPU accelerated MPE and the new 32 bits linear color processing: http://forums.adobe.com/message/3939400#3939400 but it involves more cross-fades transparency to 0.
So I'm still wondering if anybody else is experiencing these type of glitches...?!
I don't think that is your issue, since you source is avchd. Maybe new GTX drivers would help?
I'm using a GTX560 on a PC at home with no issues, and it is almost the same card.