Is there a way in after effects to preserve jpeg image quality when you save frames as files? I want to use after effects to apply the keylight effect to some pictures but when I render them as photoshop files they lose quality.
Your Photoshop file will be the size of your composition and if you render, the size of the render settings. Make sure the settings are correct. I don't know what you are expecting but video, and therefore an AE comp is awfully small compared to a modern DSLR or even a new smart phone photo. Some more details would help us help you.
but when I render them as photoshop files they lose quality.
unfortunately the default of saving a frame is to use "current settings" and not "best settings". this means that if you resolution pop up is set to other than "Full", you will get reduced quality.
so make sure it's set to "best settings" and do as I do, change the frame default to "best settings" so it wont bite you next time. you can access the settings in edit->templates->render settings