11 Replies Latest reply on Feb 26, 2016 2:46 PM by ZakIckes

    Roto Brush tool fails me everytime, why?!

    MarkNotton Level 1



      My main reason for getting After Effects CS5 was to take advantage of this new and "amazing" roto brush tool.


      Whilst the princible and the examples I've found online make out that this tool can do the impossible... I find it just doesn't work.


      I've roto-brushed/masked around my entire video clip, I've scanned through it all and it all seems to look perfect. I then add to Render Queue, and export it. And that's where the Roto Brush fails. It simply screws up all my work by adding lumps and bumps that were never there.  It seems to re-render all the roto-brushing I've done and messes it up completely. I've tried exporting to Quicktime, AVI, and even PNG Sequence.


      Is there a way I can export the video clip EXACTLY how I've roto-brushed it? Can I stop Afert Effects from trying to be too clever, and prevent it from re-doing what I've all ready done manually?


      On a side note - What is all this "Propagating" stuff? After using the Roto-Brush, AE always needs to "Propagate" the entire clip everytime I scan through the timeline. I'd have a little more patience if I knew what it was actually doing.




      Mark Notton

        • 1. Re: Roto Brush tool fails me everytime, why?!
          Todd_Kopriva Level 8

          Here's a page that collects some resources that teach you how to use the Roto Brush tool.


          My first guess about why your results are different when you render for final export is that you aren't working at full resolution in the Layer panel. Here's a tip from the official documentation for the Roto Brush tool, which you really must read before trying to use it:

          "Work with resolution set to Full when using the Roto Brush tool. Fast Previews modes, such as Adaptive Resolution, don’t work well with the Roto Brush tool, because switching resolutions requires a full recalculation of the segmentation information."


          That same page also answers your question about "propagation". But the basic idea is that After Effects uses the strokes that you make on one frame to decide what to do on other frames, and propagating (i.e., spreading the information through time) takes some time.

          • 2. Re: Roto Brush tool fails me everytime, why?!
            MarkNotton Level 1

            Thank you!


            John Dickinsons tutorial explained the two main topics I needed. I understand Propogating now, and how the Freeze RotoBrush works.


            I got a little worried at one point, because the intructions on the official Adobe help site were of no use. Only because I had already followed those intructions before-hand.


            Thanks again, After Effects has redeemed itself! (to me anyway).


            Mark Notton

            • 3. Re: Roto Brush tool fails me everytime, why?!
              Todd_Kopriva Level 8

              I'm glad that it helped.


              BTW, before you write off the "official Adobe Help" document... I wrote that, and I'm also the person who commissioned John Dickinson and Mark Christiansen to create their materials, precisely so that I could point to them from the Help document. We work with these outside folks to create these tutorials and supplementary materials because we know that they help a lot more than reading a dry text description. The dry text description has its place, too, because it comprehensively contains all of the deep information from one of the folks who designed the feature (me). You might not need all of that information at first, but it's good to know where it is when you have questions (like exactly what's happening with the propagating).


              My point is that you are almost always best served by starting on the official Adobe Help page for any feature (or the main Help & Support page) and then checking out the tutorials linked to from there. For the big new features, you can also check out my blog.

              • 4. Re: Roto Brush tool fails me everytime, why?!
                MarkNotton Level 1



                My apologies, I didn't mean to suggest that the Adobe help page was "generally" of no use. Only that I had already followed those tips previously; and yet continued to have the same problems.


                I did however overlook the functionality of the Freeze button, which in hindsight is pretty self-explanatory. It was the video tutorial that made it clear. I guess I still have a stubborn nature to just experiment with buttons and options, and hope it just works... as opposed to patiently reading a few paragraphs of text.


                Lastly, I'd like to thank you personally for the exceptional efforts gone into developing the feature. You have saved many man hours, and provided people with tools that will massively improve standards of work. Thank you


                Mark Notton

                • 5. Re: Roto Brush tool fails me everytime, why?!
                  Todd_Kopriva Level 8

                  Again, happy to help.

                  • 6. Re: Roto Brush tool fails me everytime, why?!
                    ZakIckes Level 1

                    Ok. So I've read your help page. I've also watched the tutorials, both from Adobe Lynda and other professionals. However, this feature is broken, actually After Effects CC (especially mac) is broken. I spent days roto brushing a minute of complex footage. I then moved on to roto brushing additional segments of footage, eventually finishing three separate movie layers that spanned a collective 3 minutes. When I decided to add my additional visual effects, all of the mask for the first clip were gone. It was like I never even touched the footage (except for the roto brush propagation path in the layer window for the mov). There was no matte left over.

                    I used the layer window, I froze the clips when finished, full resolution, I did everything right. The worst part is as I tried to investigate, my whole project began failing on me. I reopened the project, and the first clip was permanently corrupted, all of my roto brush strokes (although still present as keyframes) had visually disappeared minus several (?) random ones. I closed the project, and opened up a previous save. To my luck the original roto brush stokes I had drawn a week prior were still there. I re-opened the newer project file, and imported the correctly roto brushed layer from the older project. To my horror (although not my surprise) this didn't change a thing. Somehow the correct layer was changed in between me closing the older project, and opening the new one.


                    So what gives? I've been on one system, and the footage has not changed location. Before we try to blame my spec's I have been doing some more intensive work with fewer issues. Not to mention everything has been going along fine until tonight.

                    • 7. Re: Roto Brush tool fails me everytime, why?!
                      Rick Gerard Adobe Community Professional & MVP

                      First, rotobrush uses a ton of resources so you should limit the length of the clip to just the part that will be used in the final edit.


                      Second, if the original footage is not in a production format or an image sequence but is instead compressed original from a consumer or even a prosumer camera then AE will have to decode the footage and turn it into frames before rotobrush can do it's work. You will be better off converting the footage to an image sequence or 10 bit or better production format before you try and use rotobriush.


                      Third, your sample frames look like some kind of hand drawn animation. The likelihood that this type of footage will successfully be traced and the edges refined is very low. I can't tell from the examples what kind of background you are trying to remove.


                      Fourth, as soon as you get a successful progression of the Rotobrush tool you must FREEZE the analysis then render the footage layer to a suitable digital intermediate that supports an alpha channel. Unless the shot is just a few seconds long it's nuts to try and use the shot in another comp or use it through Dynamic Link in Premiere Pro.


                      Last point. Once you have a successful root and render your DI you should delete the rotobrush comp or the effect to free up system resources and reduce the size of your project. I have never found it to be a good idea to leave rotobrush active in a project. There's too much that can go wrong. The same thing goes for Warp Stabilizer. Apply Warp Stabilizer to only the portion of the shot that you are going to use in the final production and then render a DI and get rid of the Warp Stabilizer effect.


                      Both of these tools are still in their infancy. They both save time but neither of them work well without good preproduction planning.

                      • 8. Re: Roto Brush tool fails me everytime, why?!
                        ZakIckes Level 1

                        Rick absolutely. If I had the time, system and budget to properly transcode all my footage I would, and I usually do. I also want to point out the hand drawn footage is not mine... I will admit in this situation all your points are very fair. However, my problems are more than just rendering issues, they are definitely bugs. And multiple of them have happened. I could list them and post screen shots, but why bother. They have been touched upon on other forums, and almost all off them have little to no answers.

                        One thing I would like to bother you about is saving video files with an alpha channel. I originally attempted it, but could not figure it out, so I continued rotoscoping. Is there a specific file format that plays nicely with AE and supports an alpha channel? This work around would be huge.


                        And my problem isn't with the tools, It's the lack of a realistic disclaimer. I get one has to sell their product, but no roto brush tutorial goes over specifics for file format and the such. You only get told to be at full resolution, same frame rate etc. Adobe built this tool and they should in plain text say if you want to use EFFECT A note that you should use SETTING A, SETTING B, SETTING C - or else our program will crash and potentially corrupt itself forcing you to go to a slightly older save. My problem was the program unfroze my roto brush strokes, and then erased their positional information. The only way to get that 30 seconds back was go to a previous save, but like you said importing that kind of information is nuts.


                        And despite my tone, I really appreciate your help. I'm just mad at the machine, and myself.

                        • 9. Re: Roto Brush tool fails me everytime, why?!
                          Rick Gerard Adobe Community Professional & MVP

                          The output module in the render Q has a preset called lossless with alpha.

                          Screen Shot 2016-02-25 at 11.00.54 PM.png

                          Use that or create your own by setting the output module Channel settings to RGB plus alpha, the color mode to straight alpha and pick QuickTime JPEG 2000 or QuickTime PNG, both of which support alpha channels and both Windows and Macintosh. You will have to pick the codec before you can set the color depth to RGB plus Alpha.

                          Screen Shot 2016-02-25 at 11.02.04 PM.png

                          You could the Edit>Templates>Output Module to create your own custom template and set up the default options for the Output Module.

                          Screen Shot 2016-02-25 at 11.05.08 PM.png

                          • 10. Re: Roto Brush tool fails me everytime, why?!
                            Rick Gerard Adobe Community Professional & MVP

                            One more note. The only time I have ever had the Rotobrush tool fail to render the mask I previewed was when I failed to Freeze the mask. You may have done that but your screenshots are exactly what I would expect if I had forgotten to apply Freeze to the roto before I rendered.

                            • 11. Re: Roto Brush tool fails me everytime, why?!
                              ZakIckes Level 1

                              Aha that seems to work. I believe I failed to check the color management page, that would make sense. Thank you for the great pictures, not the mention the extra tip for saving export presets A+. This definitely solves most of my issues, thank you.