Well, you get what you ask for - compressed footage. Tweaking the settings can possibly improve things a bit, but there will always be artifacts, especially with screen capture footage which may have large static areas of uniform color, which is pretty much the exact opposite of what these compressions are based on. They expect motion and make it part of their perceptive evaluation to disguise the nature of their compression. Jerkiness is a whole different matter, but you may wanna check your frame rates. For everything else, provide more detailed info.
Alas, the results you are seeing, Banding, is an artificact of the underyling compression process used by FLV and F4V compression.
See the section on Banding here: http://www.estadistica.cl/public_disk/Programas/DeTodo/Manejo%20y%20pegado%20de%20subtitul os/VirtualDub-1.6.14/help/p-artifacts.html.
Here's a good video about it and some workarounds: http://greyscalegorilla.com/blog/2009/10/how-to-remove-banding-artifacts-in-after-effects/