What codec are you using to export to or are you exporting with "None" compression in Quicktime? If you're using "None" try using the Animation codec sliding down the quality until you find it completely acceptable from a quality standpoint but allowing it to compress the file a bit.
Do you have much space left on the drive you're running Flash from? Typically this would just go into swap memory rather than run out.
How much memory do you have? What kind of video card do you have? Low end video cards (not typical on Macs) can use system memory as video memory which can get gobbled up by GPU utilizing apps (especially Photoshop, a real memory user, both GPU and system). In fact if you just have Photoshop open and 4GB ram or less then having Photoshop loaded alone is probably your issue.
Experiment with other codecs as well. They all use varying amounts of RAM to render.