I have the same system. Make sure in preferences you just have the computer monitor selected as the output, not the Kona. AE has always exported to the Kona easily. I had the same problematic playback when I made the mistake of selecting the Kona and not just the computer monitor.
BTW. Im lovin my Z820. I also have 64MB ram, but opted to add the GTX 570 for better performance than the cards they supplied .Also a Dulce DQg2 16TB raid that has 1100/1000 MB/sec. read/write speeds.
However, I do think I have playback set to computer monitor (not in front of system right now). Definitely seems like there's something AE doesn't like about my card (it is an older version of the Kona 3). I have a support ticket in with AJA so hopefully that will turn up something.
As far as my Quadro 4000 goes, I was kind of forced into using it on account of other software on my system. If it was up to me, I would have put in a GTX680 or 690. I have a GTX680 at home and it is rock solid in AE and Premiere. Timelines that are heavily dependant on GPU accelerated FX render about twice as fast on my 6-core/GTX680 vs my z820/Quadro 4000 work system. I'm hoping to upgrade to the Quadro 5000K when it comes out, possibly with the Tesla sidekick. In the meantime, I have to sort out this Kona thing
I also have a 4 year old Kona but not the same issue .
Just for grins, have you tried uninstalling the AJA drivers and see if your playback is affected. That would be my next step in the trouble shooting process, before I sent the card away to AJA.
I take it PrP is running just fine with with Kona and its just AE that is the problem.
It just sounds EXACTLY the problem I had when I first setup the system monitoring in AE Prefs and had the wrong thing selected.
Okay, I'll double check tomorrow am, but like I said I'm 90% sure I have it set to computer only. As I recall, playback was smooth when I was first setting up the system (before Kona and other third party cards were installed). Definitely need to start pulling some cards to troubleshoot. Were you also getting sporadic signals to the Kona or was it just stuttery playback?
It just wouldnt play at full speed until I made the monitor assignement change.