I followed the tutorial on CUDA and my preferences would accept my GTX 780m as an approved card. I forced the unsupported card for CUDA and this is a copy of the info, there is are no options to use OpenGL in the Advance section of the Cartoon Plugin :
Fast Draft: Available
Texture Memory: 1585.00 MB
Ray-tracing: GPU (unsupported)
Unsupported GPU enabled for CUDA acceleration
Vendor: NVIDIA Corporation
Device: GeForce GTX 780M/PCIe/SSE2
Total Memory: 3.87 GB
Shader Model: 4.0 or later
Driver Version: 6.5
Devices: 1 (GeForce GTX 780M) (unsupported)
Current Usable Memory: 3.62 GB (at application launch)
Maximum Usable Memory: 4.00 GB
The OpenGL stuff has nothing to do with the CUDA settings. Problem probably is that it can't handle your card. Perhaps it's the memory, since openGL capped off at 2GB, perhaps a driver issue. Impossible to tell.
"Rewrote Cartoon effect so that it doesn’t prevent the use of Render Multiple Frames Simultaneously multiprocessing. The Cartoon effect also no longer uses GPU acceleration."