while AE is open its not possible to use nvenc on any other program. disabling graphics acceleration in AE doesnt help. only closing AE helps using other tools. whats the problem?
Nobody can tell you much without exact system info. Generally, though, that's a limitation of most GPU-acceleration models - only one "client" can have full control over the GPU, especially with deep access like CUDA/ OpenCL that takes a firm grip over the card's memory registers.