Disable the QT plugin for the Matrox stuff. It clearly uses a memory area in your video RAM which is conflicting with normal display operation.
Sorry for the ignorance but how would I do that?
Go to the Library/QuickTime folder on your hard drive and move the Matrox files. (This is the Library folder on the root of your drive, not the one in your user folder.)
I get this same error with a Blackmagic Multibridge on Windows Vista x64 if I leave the Blackmagic component in the (Windows equivalent) QuickTime folder. I suspect either the Blackmagic 64-bit driver is misbehaving or QuickTime just isn't playing nicely on Vista x64.
Here;s another. I have Quicktime Pro installed. I have CS3 installed. I also have AE v7 which runs a few plugins I am still thinking work in v7. CS3 runs OK but in version 7 it gives me the "Couldn't open Quicktime Video Output Component " notice. I had a client working in v7 so I tried to work in v7, it was a bad idea. I lost days of work and finally just went to CS3 to get the job done but couldn't use one anothers files between versions.
What can I do to get the AE v7 Couldn't open Quicktime video output component fixed. Make it simple if you can... I'm an artist.
In your case there is nothing you can do, most likely. Apple have stopped supporting specific legacy Quicktime function calls a while back and newer versions of QT have become incompatible with older programs because of that. You could only downgrade to an older version of QT. It's also possible, that a clean re-install of the whole operating system and then using a standalone install of QT may work to straighten things out. In any case: you should let go of AE 7 and instead update your plug-ins. Things will only get worse the more we approach Snow Leopard....