Well, if the system disk is really that full, then that would be more of a general system issue, I'm afraid. RB as such doesn't do scratch disk. All that is handled by the system swap file (if it runs out of RAM) or the normal AE disk cache. You will need to develop an alternate plan. If you have a second drive, setting the system's tmp variable/ virtual path to that drive may already hugely help. Likewise, moving the AE caches to that disc and cleaning them out might improve matters...
I think you're right that it's the system swap file that's filling the drive. My startup disk only has about 7gb free at the moment. (I only use it for applications, and I run almost everything else off of additional drives. The first thing I did when I installed CS5 was to change the cache locations.) I've never had trouble before -- CS4 seems to just stick everything in the AE cache -- but roto brush in particular seems intent on using the system swap file instead.
Is there a way to force it to use the AE cache path, or to change the swap file location without dealing with the command line? Right now it sounds like the simplest solution is just to move a bunch of stuff off my startup disk and hope that's enough to keep it from crashing.
I have an 8-core mac pro (OSX 1.5) with 8gb of RAM (1gb/CPU allocation). I've reserved 1 processor and 1.5 gb ram for other apps, but the only other stuff I'm running is email, IM and a couple browser windows. All my cache settings are set to non-system drives.
I cleared another 35gb off my system drive and it seems to be working now, but the longest clip I'm dealing with at the moment is only 15 seconds or so.