It is very difficult to understand what your problem may be without seeing or understanding precise structure of your project. We need a screenshot that shows the project and reveals all of the changed properties on the layers you are having problems with. Because your problem involves a pre-comp a flowchart would also help.
I found the mistake which cause the problem.
The second I use layer styles in the precomp, I get this problem. can you explain why??
Layer styles render in a different order so lots of things can get fouled up with you use them. They are still usable in complex situations, you just have to apply them in the right place in your composite. I've never found a situation where I needed a Layer Style and another effect that I have not been able to work around the limitation.
There are other things that change the render order. After Effects only does one thing at a time. The basic render order is listed right in the timeline view. Here's a sample.
Basic effect with Layer Style turned off:
First Mask 1 is applied to the layer, then Mask 2 is subtracted creating the hole
Then Fast Blur is applied, then Bevil Edges to create the button with the soft edges
Then the Transform controls take over and the anchor point is evaluated, the position reported, then the scale, then the rotation, and finally the opacity
Then the Layer Style is applied.
In my button design I want to apply a stroke to the button so that I looks like this but with the soft base and a stroke on the inside hole:
But, as you can see I had to turn off the Fast Blur to get the stroke to work because the Layer Style Stroke is calculated from the alpha channel, which fast blur has made 76 pixels wide so with Fast Blur on I get this:
The solution to my problem is to pre-compose and add the layer style to the pre-comp, then use an adjustment layer with masks for the fast blur to get what I wanted in the first place:
So here's what AE is doing with the final project.
- The background layer is rendered
- The pre comp is added to the mix
- Inside the pre-comp the layer is rendered
- Mask 1 is added to give rounded corners
- Mask 2 is subtracted to punch the hole
- Layer styles are applied to add the stroke
- Back in the main comp the pre-comp is rendered
- Bevel Edges is applied to the pre-comp using the alpha chanel
- Now the adjustment layer is added to the mix
- Mask 1 is added and this limits the area in the underlying layers where effects applied to the adjustment layer will work
- Mask 2 is subtracted from the adjustment layer poking a hole in the effects
- The fast Blur is added blurring only the bottom edge of the button
There you have it. An in depth tutorial on the rendering order in AE and how to use nesting and pre-comps to reorder effects to get what you want.
Thank you very much Rick for your time and the really great explanation.