This has nothing to do with the source format. AE is completely agnostic in that regard. DV is heavily compressed, that's why. You are actually seeing the blocks of the compression. Your other camera may use a less agressive compression (AVCHD, MPEG-II I-frame only or whatever), hence it will work better for keying. It is however absolutely possible to get good keys even with DV, it just may involve additional steps. This takes too long to explain, but if you search this forum, you will find enough tips and tricks provided over the years. You can also find enough tutorials on the matter on creativeCOW, videocopilot.net and other sites, so ask the mighty Google.