Well, read up on what your VJ software actually expects. In addition to the Animation CoDec, TGA, TIFF, PNG or PSD image sequences all retain transparency.
I am not aware of any software that can play Animation in real time. Usually requires proprieatary codec and hardware support. ProRes 4444 is a good choice for Apple-based products but, as Myl says, your VJ app will have specific requirements that spelled out in the documentation. Let us knwo what those settings are and we will assist you in ceating AE output that will work.
I got an animation with a transparent form on blackground, someone told me that if i export the animation with ".mov / animation compression" that will be ok and i keep the alpha layer (transparent form) in my vj software... but that fail...
I'm not surprised. You only got a partial answer. You created NO transparency following those instructions.
Add your comp to the AE render Queue. Open the Output Module. Select Quicktime Movie, Animation codec.
Now here is the information those geniuses didn't give you: Select RGB+Alpha and Millions of Colors +. You will find an additional setting: Straight or Premultiplied. I suggest you render both. Use AE's Add Output Module command, set one of the renders to a Straight Alpha Channel, the other to the Premultiplied alpha channel.
Oh, and don't forget to give the files two different names.
One of them will work, and then you will know which kind of alpha channel you need to render in the future.