I'm working on a project doing keying 4k dpx's in log. However, I noticed that if I use keylight to key the log images, after I convert them to linear color, they show hard ugly edges (they look like horribly exaggerated edge color correction, bright orange on people's skin) to the key. However, if I convert to linear before doing the key, they look fine.
If I have to, I'll do all the compositing work in linear and convert back to log at the end, but I think this will result in some generation loss. Is there any way to make keylight understand log?
Not that I know of. It goes against how keyers work and I beleive even in Nuke and other programs the respective nodes do an automated conversion to RGB or YUV when necessary. You should not experience any generation loss in technical terms since you will be working in 16bpc or 32bpc, but of course the colors may shift around. Therefore you may wish to use color profiles. There's a generic DPX profile included in CS3.