It depends on which applications you use and how much compatability you need with older versions. There is no need to remove CS4 other than it might free up a small amount of disk space.
There is also a good chance that the file associations for different Adobe documents will be affected after the removal of CS4.
Is it really "just a little space"?
I don't know what CS4 product you have installed so I could not tell you how much space is currently being utilized. My guess is that you are looking at 1 to 3 gigabytes.