There are a few things going on with the default version of Catalyst CS5 that might make it respond more slowly (especially on a big project - lots of nested components with many states):
1. The binary that runs catalyst is still 32 bit so on Windows (Even a 64 bit system with 12GB of RAM) it can only use 2GB max
2. Since the process that's being run uses a Java Virtual Machine to run (JVM) it can only use 1.2 GB of that theoretical max of 2GB
3. By default it comes configured to only use 512MB of heap in the JVM, but this can be modified as shown in the link below:
Theoretically you could run it on a 64bit JVM that takes advantage of more than 2GB of RAM per process, but I am unclear how that works with a compiled .exe -> JVM (Tomcat had a similar issue a while back before they started including a pre-compiled 64bit Bootstrap loader for windows)
We were having issues with an extremely large Catalyst CS5 project and by doing the modifications linked above it gave us a bunch more breathing room.
Another reocmmendation: Upgrade to Catalyst CS5.5 - opening the same project in Catalyst CS5.5 (without the .ini modifications) was extremley performant so it looks like they have done some other performance enchancements between Catalyst CS 5.0 and Catalyst CS 5.5.
Hope this helps.
p.s. It looks like Adobe is getting on the bandwagon of 64 bit as some of their projects (Photoshop and Premiere?) have already been converted to take advantage of 64 bit. Being a developer myself I understand the implications of a "simple" change like converting an application from 32bit to 64bit.