Joerg, show a screenshot of the exact error message that you are getting, and explain in detail the circumstances under which it occurs.
That message has nothing to do with RAM. It is about VRAM, the memory used by your GPU. It is telling you that the GPU doesn't have enough memory to use one feature, the ray-traced 3D renderer. You can ignore this if you're not using that feature.
What GPU do you have? Did you follow the instructions in the message and close other programs before restarting After Effects?