4 Replies Latest reply on Jan 18, 2015 2:05 PM by briansilva

    Use roto brush result in another layer




      I've used the roto/refine edge brushes to isolate an FG object in some shot footage. I'd like to take the resulting mask from that and use it to cut out a few 3D layers that should appear to be behind it. Is there a way to export the roto brush result as a mask, or use the roto brush effect on another layer? Or even export just the alpha into images that I can pull back in as a footage layer?


      The only solution I can think of right now is to copy the footage layer with the roto brush effect and layer the cut-out FG element over the 3D elements, but I'd prefer to hold out the 3D elements rather than add an extra layer.


      Disclaimer that I'm fairly new to AE, though I'm experienced in other packages.


      Thank you!

        • 1. Re: Use roto brush result in another layer
          Mylenium Most Valuable Participant

          Disclaimer that I'm fairly new to AE


          No need to, we would have known, anyway.... Of course there is any number of ways to render clips with Alpha (only). Otherwise one would simply use the pre-composed Rotobrush as a track matte. Well, whatever, clearly you have some learning to do.


          Getting started with After Effects (CS4, CS5, CS5.5, CS6, & CC) | After Effects region of interest



          • 2. Re: Use roto brush result in another layer
            briansilva Level 1

            I'm mostly interested in a way if doing it without duplicating things. I'm using my original footage layer to get the alpha with the rotobrush, which puts the Roto Brush effect on that layer. But I don't want it to affect that footage, so I disable it.


            To use a track matte, I'd need to have the roto brush effect enabled. So I need to have two layers with the footage: one at the bottom w/o the roto brush effect, and one with it enabled above my pre-composed 3D layers set to use it as a track matte.


            If I render out another clip with just the alpha generated by the roto brush, then I have the alpha in one layer and the effect on a second layer, so there's duplication.


            I'm looking for a way to refer to the roto brush alpha directly from another layer, if this is possible, while it's disabled on that layer. Let me know if I'm missing something. If duplicating/exporting is the usual AE workflow, that's cool, but that's really what I'd like to know.

            • 3. Re: Use roto brush result in another layer
              Rick Gerard Adobe Community Professional & MVP

              There's no rendering hit with two copies of your roto masked layer. There's no reason not duplicate the layer and make it a track matte. You must have a source for the alpha somewhere. If you turn off the RotoBrush effect it is just the same as never applying it so there's no matte to extract. I would duplicate the footage layer first, then roto the top copy and use that as a track matte.


              If you really don't want to duplicate the layer but still want to have access to the matte info from Rotobrush you can use other techniques to re-arrange the rendering order so that the roto mask is still usable but the footage does not appear to be masked. This will put only two layers in your main comp.


              Pre-compose your rotoscoped footage moving all attributes. This puts the mask at the top of the rendering order. Apply Set Channels to the Pre-composed Footage layer and set Alpha, the bottom option, to full. This leaves the first instance of the alpha in the rendering order stack. You can then use Set Channels or Set Matte on the layer you want to apply the roto matte to and extract the alpha from the top of the rendering order. This way the Alpha is extracted from Rotobrush before the Alpha is set to full by Set Channels. That is the only way to have only one copy of your footage in your main comp, but it would take just as long to render as simply using a duplicate and track matte.


              Another option would be to RotoBrush your footage then render a copy with alpha and then apply Set Matte, or render an alpha only copy of the footage to use as a track matte.

              • 4. Re: Use roto brush result in another layer
                briansilva Level 1

                Thanks Rick, that’s wonderful info!


                The Set Channels/Matte approach is cool, and I will make use of those in the future I’m sure. What you suggested worked great for me (though, curiously, if I set the down-sample to “quarter” in the composition viewer, setting the alpha to “full” stops working… is that a bug?), and it’s a good exercise in understanding layer render order.


                But if the duplication workflow is efficient and a common approach, I’ll stick to that — it is straightforward.


                Thanks again, I appreciate the help!