You are right that the graphics card doesn't matter much for After Effects. The stock GPU in any modern computer is enough. The specs that you mention for other components seem fine for running After Effects. Premiere Pro is a different matter; it uses the GPU a lot more.
See this FAQ entry for more information: