First off, which CPU do you have on that motherboard? If you have something that's very low end, such as a Pentium G6950, you're pretty much performance-limited: One such system on the PPBM5 list attained a result of more than 1800 seconds (and that's with an Nvidia CUDA card that was not configured with the MPE hack) - more than 35 times slower (RPI-wise) than a fast i7 system. That's because the G6950 is a dual-core CPU with no HyperThreading, and is thus limited to only two threads. But if it's an i7-8xx CPU, you might be able to get away with upgrading to a GTX 550 Ti or GTX 560. If it's an i5-750 or 760, IMHO it would be somewhat of a waste of money going for anything higher than a GT 545.
I have a 2.8Ghz Intel Core i7-860 8MB Cache Quad-Core The card I am going to replace is a 1GB ATI Radeon HD5770-GDDR5 PCI-E Dual DVI. How should I address the aditional HDMI monitor to view the video being edited also?