The Dell T1650 units are the Desktop/E3 Platforms. Those max at Quad Core and 32GB of ram. A Geforce 660Ti would likely be the best card you could use or need with that system. That is what I would suggest.
In addition to what Eric stated, you will need to purchase unbuffered ECC RAM (or ECC UDIMMs) for that Dell Precision T1650: The chipset used on the motherboard requires ECC RAM, but none of the LGA 1155 Xeon CPUs support registered memory.
To add some more details to Eric's advise, discern between various configurations and keep in mind that this system can not be overclocked.
First, possible and sensible options for video cards for such a system comprise:
- GTX 650
- GTX 650 Ti
- GTX 650 Ti Boost
- GTX 660
- GTX 660 Ti
Anything less is only with GDDR3 and too slow, anything more is overkill on a non-overclocked system.
With less than 16 GB memory, only options 1 & 2 make sense. With 16 GB the sensible options are 2, 3 and 4 and options 4 & 5 only come into the picture with 32 GB memory. If you do no resizing of source material on export, for instance AVCHD to BD-R at 1920 x 1080, you can choose the lower numbered option, if you do rescaling or frame blending regularly, changing from 1920 x 1080 to some other resolution, go for the higher numbered option. Same applies to shooting in 24p and wrapping the export in 29.97i, go one higher. If your disk setup is weak in terms of number of disks, remember that hardware MPE only assists in certain cases, extra disks all the time, so you may be better off saving some on the video card and investing that in extra disk(s).
You state that you are jumping ship because the iMac was not upgradeable enough but you are thinking about a T1650!? That is a really limited box so far as upgrading goes: small power supply, limited drive bays, etc. If you are resolved to go with a Dell + third party parts, you should consider the T3600 as a minimum (much stronger p/s, more RAM options, can take 6 and 8 core cpus, etc.) or possibly something from Dell's gamer's line of PCs. You are already planning to upgrade with 3rd party parts, so maybe you should consider building your own system that is truly tailored to the task.