13 Replies Latest reply on Jan 17, 2012 6:44 PM by Rick Gerard

    Problems with creating a mask in AE 4.1

    digiday Level 1

      Hi all,

       

      Yes, I'm using After Effects 4.1... not CS4... that's version 4.1 from 13 years ago.

       

      I'm having a problem creating a simple mask and I can see from a more current AECS5 tutorial that creating a mask on a layer is similar to creating one in AE4 based on a comparison to an old AE4 tutorial.  I've read instructional material and it seems simple enough, but for some reason its not working for me... I know I'm doing something wrong.  I'm an advanced Photoshop user and I know the concept of layer masks and how to create them.

       

      I create the mask on a layer and I can see and adjust the bezier points to adjust the shape of the mask... adjusting the shape of the mask by the bezier points is a little tricky since it stretches in both directions when you grab a single point, but that's okay,

       

      But no matter which mask mode I try, Add, Subtract, Intersect, Lighten, Darken or Difference, it is not masking the layer that it is attached to even when I preview the timeline.

       

      I don't want to just create a color layer on top of the layer I want masked to simulate a mask (which I can otherwise do) because I want to stack layers with masks and have the unmasked parts showing through (like in Photoshop), without rendering separate layers and re-importing them with blue screening or black screening... I should just be able to mask the parts that I don't want to show up on each layer and only have the unmasked parts of each layer show through each other for my final render.

       

      Thanks for any input,

       

      digi

        • 1. Re: Problems with creating a mask in AE 4.1
          Dave LaRonde Level 6

          I'm wondering if you're masking the right way for the software version.

           

          In AE 4.1, I'm pretty sure you have to double-click on a layer to open it in its own window, and THEN mask.  Then you close the window to see the comp window.  It's kind of klunky, but hey: we got work done with it.

           

          And the masking modes work just fine.  Or at least they should.

          • 2. Re: Problems with creating a mask in AE 4.1
            Rick Gerard Adobe Community Professional & MVP

            The procedure for creating a mask using the pen tool has always been the same. The default mode is ADD. You just have to close the mask.

             

            Create a new solid layer. Use the pen tool and click around the image until you get a shape. When the mask closes only the area inside the shape remains.

             

            Go through the basics of AE here. It's quite straight forward. All of the basics, setting up a project, masking, setting keyframes, haven't changed a bit.

             

            Here's a quick demo. Works in AE 3, 4 all the way to CS5.5.

             

            • 3. Re: Problems with creating a mask in AE 4.1
              digiday Level 1

              Thanks for writing back Dave and Rick,

               

              I think there is something in what Dave said about opening a new layer window, but it still isn't working right.  Please see the screenshot of two full project windows below.

               

              One thing that could be giving me problems is my layer content... it is a numbers layer attached to a solid layer, as you can see.

               

              In the comp window at the top of each screenshot below, you can see the bezier points from my original mask that I created by clicking on 'Shape' next to 'Mask' on the left in the timeline window after selecting 'Layer>Mask>New Mask'... but I can see that this isn't the right way to do it because (1) it isn't masking, it only shows bezier points. (2) When I try to adjust those bezier points in the comp window it only stretches the numbers. And (3) when I'm in the comp window, I don't have access to the layer shape square and oval tools... they are grayed-out.

               

              So in the layer window after double-clicking on my numbers layer, in screen #2 below, you can see that it isn't showing anything... just black, but you can see the bezier points from my original masking attempt that also show up in the comp window.  But in that layer window I do have the square and oval tools available to me and you can see that I tried creating a new experiment with a triangle mask, but that only shows bezier points and no masking effect.  When I close that window, as Dave said, after making the new triangle mask, it only broadens out my original mask besizer points in the comp window to encompass both masks now, but does not reflect either masks actual shape, nor is it actually masking anything.

               

              Let me know what you think.

               

              Thanks again,

               

              digi

               

              MASKINGproblems.jpg

              • 4. Re: Problems with creating a mask in AE 4.1
                Dave LaRonde Level 6

                Oh, now I get it!  Because of the effect you're using, the mask is ineffective.  You know about track mattes?  Use a solid as a track matte for the digital clock layer to conceal the unwanted digits.

                • 5. Re: Problems with creating a mask in AE 4.1
                  Rick Gerard Adobe Community Professional & MVP

                  You can also pre-compose the layer and add the mask directly to the pre-comp.

                  1 person found this helpful
                  • 6. Re: Problems with creating a mask in AE 4.1
                    Dave LaRonde Level 6

                    Yeah, and if you want to re-familiarize yourself with the way masks work in AE 4.1, just use a plain-old solid, with no effects applied to it.  You'll have a lot better luck.

                    • 7. Re: Problems with creating a mask in AE 4.1
                      digiday Level 1

                      Okay, I get it.

                       

                      So I can't do a plain mask on a numbers or letters layer... but I can...

                       

                      1) Do a track matte with a white mask layer on top set to Chroma, and a second layer with the numbers set to track matte so only the area of the white top layer shows through... I've done that kind of thing in Premiere successfully to get picture in picture and stencil mask text on an animated background.

                       

                      2) Render the numbers layer to an AVI and then import it in to a new project and mask it there.

                       

                      3) Just put black layer squares over what I don't want, which masks to black, render that and then layer the rendered files on top of each other using black screening to bring them all together.

                       

                      Okay, thanks for clearing this problem up... I couldn't find info on this limitation anyplace else.

                       

                      Thanks,

                       

                      digi

                      • 8. Re: Problems with creating a mask in AE 4.1
                        Dave LaRonde Level 6

                        Track mattes haven't changed one whit since the days of AE 4.1.  You could look at AE 10.5 Help, search for "track matte" and find the info there useful.

                        • 9. Re: Problems with creating a mask in AE 4.1
                          digiday Level 1

                          Thanks Dave,

                           

                          I've used track matte techniques previously in Premiere, as I mentioned in (1) in my last post.

                           

                          Here is a sample of the use of track matting to create the base animated text with Premiere in this short video I created late last month...

                           

                          http://www.youtube.com/watch_popup?v=f-sUWa9FGdQ&feature=youtu.be&vq=hd720

                           

                          digi

                          • 10. Re: Problems with creating a mask in AE 4.1
                            digiday Level 1

                            Hi again,

                             

                            I just want to check to see if this is right.  When I am masking on a normal layer in AE 4.1 (not on a numbers or text layer), when I'm in the layer window to do my mask creation, I still can't see what I'm masking over?... is that correct?  It appears that I'm masking over solid black no matter what.

                             

                            It seems that when I'm creating the mask with the pen tool or with the square or oval selection tools in the layer window, I am working blind and I'm not able to see what I've masked until I close the layer window and look at the comp window.

                             

                            Then if the mask isn't placed or shaped correctly, I need to go back to the blind adjusting of the mask in the layer window again, then close that and see if the mask is correct in the comp window... right?... or am I still missing something.

                             

                            Thanks,

                             

                            digi

                            • 11. Re: Problems with creating a mask in AE 4.1
                              Rick Gerard Adobe Community Professional & MVP

                              Watch this simple movie:

                               

                               

                              Keyboard shortcuts used Cmnd + N (new composition), / .,,.. (zoom in and out of composition window), Cmnd + Y (new solid), g (select pen tool), m (reveal mask path in timeline). That's all there is to it.

                               

                              The only thing you may be missing is that you may have vidw masks turned off in the Composition window. It looks something like this:

                               

                              Screen Shot 2012-01-17 at 5.06.28 PM.png

                              • 12. Re: Problems with creating a mask in AE 4.1
                                digiday Level 1

                                Hi Rick,

                                 

                                In this tutorial and the other video tutorial that you posted it shows the mask creation and editing being done directly in the comp window... that is not how it seems to work in AE 4.1.

                                 

                                If you look 11 posts up, you'll see that Dave LaRonde wrote:

                                "In AE 4.1, I'm pretty sure you have to double-click on a layer to open it in its own window, and THEN mask.  Then you close the window to see the comp window.  It's kind of klunky, but hey: we got work done with it."

                                 

                                Then I reiterated what he said in my following post about opening a layer window to do the mask editing and then closing that window to see the actual mask over the layers below it.

                                 

                                I'm not sure, but I think you may be assuming that things are exactly the same in the newer versions of AE as the old 4.1 version that I explained I'm working with.

                                 

                                But thanks for the continued posting of the tutorials for the newer AE versions... if I ever spring for the newest AE, those will be helpfull... it looks like there, the masking is as simple as it is in Photoshop.

                                 

                                digi

                                • 13. Re: Problems with creating a mask in AE 4.1
                                  Rick Gerard Adobe Community Professional & MVP

                                  I was pretty sure that editing masks in the comp window was introduced in AE 4.1. In 3.0 you had to edit in in the layer window but you could easily drag the layer window next to the comp window so you could see both. The CTI is linked so everything is fairly easy. I've been using After Effects since the Cosa Days (October 1993). I've written tutorials since AE 3.0 and It's possible that I've got my versions/features out of order.

                                   

                                  A quick Google search brought up this video. It shows how to drag the layer window next to the composition window so you can edit in the layer window and see what's going on. I also found that it was CS4 that introduced multiple masks on a single layer and the boolean operators for the masks. Take a look and see if this workflow tutorial from will work for you.