So the Filter works fine on other images of comparable size?
What are your GPU and its driver?
Does changing Photoshop > Preferences > Performance > Graphics Processor Settings > Advanced Settings (to Basic for example) have any bearing on the issue?
Only fails when processing Nikon D600 nef files. On windows vista home premium machine
Nikon D300 nef files are processed ok, all other files processed ok
All files processed ok on my windows 7 64 bit machine