Let's ask the obvious: is it an RGB Illustrator file? It should be.
And are you working in 3D, which means that the button's function changes from Continuously Rasterize to Collapse Transformations?
It is an RGB file, and I am not working in 3D.
If you have applied raster layer effects in illustrator, then it won't work because you no longer have a vector image.
What is the zoom factor of your composition window? What is the resolution of your comp window? If you would post a screenshot of your composition with the properties of the layers in question revealed by pressing the u key twice maybe we would have a clue what you are doing wrong. It's quite difficult to come up with an answer with out some details. We need more details.