I've been experimenting with creating 3D logos, text and shape layers with AE CS6 by using the "shatter" effect. I have noticed that curved letters (like "S") or curved shapes are jagged, even when rendered in full quality (see attached image).
Is this due to a setting in After Effects? How can I smooth this? I have experimented with Ray Trace as an alternative way to extrude, but with only 8GB of RAM I am unable to render out sufficiently high quality 3D text or layers in enough time.
It's just aliasing. Make your comps bigger and nest them. Make your text bigger to start with. If the layers are moving you won't notice the edges.
It's a limitation in the technique you are employing. As Rick said, make the text as big as you possibly can, then scale it down again after the effect is applied.