...I'm a Windows user,but, I do not believe that you would need to " flash to mac firmware" any newer NVidia video card you would install in your current system because the newer versions of your OS have eliminated this necessity. See the following thread :
Go to the last few pages of that thread. Assuming your OS is fully updated to the current version, it appears all that is necessary is to go to the NVidia website and make sure you are using the latest NVidia driver for your current 660ti card. This SHOULD eliminate crashing.
Next, you must insure that the 660ti is connected properly to receive all the power it needs. From the PCI generation 2 slot it is plugged in to, the card will receive up to 75 watts of the possible 150 watts it needs to run. So, an additional cable which has a 6 pin connector on each end must be connected from the socket on the GPU to the auxiliary power socket on the Mac's motherboard to insure that full power is being delivered to that video card. If it still crashes, then you would run ANOTHER cable from that GPU to a SECOND available aux power port on that motherboard, ( if any left), in order to INSURE that the GPU is getting enough power. Their are videos on youtube showing this procedure for YOUR 2009 Mac Pro.
The 660ti is FAR beyond what was originally provided in your machine for a GPU.....it draws WAY more power and is designed to run on PCI generation 3 lanes......you have PCI generation 2 lanes....not as fast, and making buying any NEW GPU card NOT worth it.
Your current card should be fine, and your immediate goal should be to get it to run properly...without crashing. Adobe CC versions of Premiere Pro have ELIMINATED the need to " add" your GPU card to a list, using a "hack" that would enable any " unofficial"GPU card. You should simply be able to "click through" a warning box that your card may not be " officially supported" and then be able to select your card for use with the " Mercury Playback Engine". Adobe CS6 and earlier versions required this "hack" to get the card running properly. An automatic program to do this hack FOR you is found at the " Studio One Productions" website, for free. If you are trying to work with 4K files, your machine MAY work with them,but, MAY struggle as a minimum of 4GB of video memory on the GPU is suggested in order to provide an adequate amount for "frame buffering", or, "frame cacheing".
Now, you appear to have a problem with the way you have set up your storage system that may create severe bottle necks, slowness, or, even crashing.
1. NEVER...I repeat...NEVER use ANY " Green" hard drives on any machine that is used for video editing,or, even other forms of production !! BAD...BAD..BAD !!!
The speed of the drive which feeds your video files,or, media to the programs in use is CRUCIAL !!!......the speed MUST be fast to keep up with the demand of the other components !!..."Green drives" create MANY problems with their SLOW rotating speed to save energy AND their " spin down " behavior that ALSO "saves energy", but, CRIPPLES the performance of your computer !!
2. Not all SSDs are alike in performance...for example the Samsung 840 EVO may have a good READ speed ,but, its WRITE speed is BAD and has caused video editors to replace them with the top performing Samsung 850 Pro series SSD.
YOUR machine is OLD and the SATA ports are limited to the OLD SATA standard of SATA II...which is 3Gb/ sec. and HALF the speed of the current SATA III of 6 Gb/sec. THIS FACTOR ALONE means upgrading your machine is almost pointless, because your SATA connected drive speed will be limited to just over 200MB/sec., when using SSDs.....unless you RAID O THEM !!
So, I would suggest to use ONE quality SSD for your operating system, programs, and any " paging file" that the OS has that is like the "Windows Page file". Crucial SSDs perform very close to Samsung Pros and are much cheaper.....the MX200 series is the current model. You should first TEST your current SSDs for read AND write speed performance. IF your current operating system would support a PCI SSD, I would maybe place one in an M.2 adapter and plug it into an available PCI slot. If this would be possible, you would place ALL of your other files on this extremely fast drive. The brand new Samsung 950 Pro will run at over 2GB/sec READ and 1.5GB/sec WRITE...at the SAME TIME !! Then, use a large ENTERPRISE level spinning hard drive to back up all your media and project files. Seagate makes models up to 6 TB which run fairly fast, due to the 128MB cache on them....over 200MB/sec !!
If their is no support for a PCI SSD drive, you MAY want to RAID 0 several cheap SSDs right off the motherboard to create a fast storage system......which ,again, you can back up with a conventional Enterprise level HDD. So, ONE SSD for OS and programs ONLY, and a RAID 0 array of SSDs for fast performance. An SSD array is FAR more reliable that one made of spinning, mechanical hard drives.
What ever you do....FIX your existing poor drive setup !! Your media, project files, media cache, cache, previews, and exports MUST be on a FAST drive to give good performance...."green " spinning hard drives will cause you to go PREMATURELY GREY !!
So in your opinion, the Adobe apps crashed not because of the GTX 660ti, but because of the low speed WD "Green" drive?
The GTX 660 ti was connected with two 6-pin power cables connected with the Mac Pro logic board, so I don't think power was an issue. And whenever the CUDA or Open CL was toggled on, the Premiere and Prelude will crash, if I use "Mercury Playback Engine Software Only" it will be fine. But I guess that sort of disabled the GPU acceleration, and thus not making full use of the the 660 ti's GPU.
I'm just hoping to be able to make more use of the graphic card's computing power.
When the Pr was rendering files, I watched datas in the Mac OS' activity monitor, and the disk read/write activity was intermittent and when it's writing it was about 10Mb/s, not even close to reach the Green HDD's own read/write speed limit (around 100 Mb/s), felt like to me the rendering did not need to read/write data in a demanding way, so I'm not sure if the fast SSD raid may help much.
There are Mac-supported PCI cards for dual SSD raids (like Sonnet Tempo SSD Pro Plus, which claimed to raid two SSDs in 6GB/s SATA speed), and I may consider get one of those cards and combine two SSDs into a RAID if the crash issues are indeed oriented from the HDD speed.
...well...it appears you have your GPU connected correctly AND you have plenty of system RAM and plenty of CPU cores. I HAVE seen other Mac users complaining that there might be a conflict between certain versions of the Mac OS and the Adobe software causing these crashes. Again, do you have the LATEST version of the OS for your Mac AND do you have the most recent NVidia driver for your 660ti video card ??
If you DO have everything " current", try placing some footage and project files on the SSDs ONLY, to test the machine WITHOUT the involvement of the " green drives" at all. See if there is any difference. Make sure you have your settings correct under "preferences" in Adobe PPro. You must allocate the correct amount of memory for PPro to use. Eric Bowen from ADK computers has said on this forum that a setting of 3GB per core is a minimum recommendation. In YOR case, that means setting a MINIMUM of 18GB for use by PPro , leaving the rest for other programs. Of course, with 32GB of memory , you can assign 4 GB per core for a total of 24 GB dedicated to PPro., leaving 8GB for other programs. If there is a setting to allow " multi-core processing" in any of your programs, make sure it is selected..
Being a Windows user I am not familiar enough to accurately diagnose your current problem. If you can contact Eric Bowen either here on this forum,or, at his company, he should be able to help you...he knows about the Mac issues and hardware.
It's 10.10.5, the most recent Yosemite, and I have the up-to-date Nvidia Web Driver and CUDA.
The RAM was set to 24 G for Pr and 8G for other apps exactly as you suggested.
I will try have all footages on a SSD and see how it goes from there.