It's counterintuitive, and kind of a longshot, but try reducing the memory you allow Photoshop to, say, 50%.
Not every plugin allocates memory from the pool allotted to Photoshop, though to the best of my knowledge Photomerge does. Whether it DID in CS3 I don't quite recall. But it's an easy test to make.
Make sure you restart Photoshop after making any changes to Photoshop's memory setting.
Virtual Memory is an operating-system configurable thing. If you have a fixed-sized virtual-page-file or a relatively low maximum then you've run out of that, and you can just go in and make it larger, unless you've also run out of space on your hard-drive, then you can add a paging file to another hard-drive if you have one.
In Vista I can get to the Paging File configuration by going through these steps:
If you have plenty of free space on your hard-drives just set it to System Managed Size. If you are running short on some drives, then allocate a paging file for each of your fast internal drives and see if it is enough.
I would suspect it is misleading you and in fact wants more physical memory. If so, you have no way of fixing that otehr than getting a 64bit machine with plenty of RAM and trying it again with the 64bit version of PS. Might work, might not. All this stitching and merging stuff in PS seems to be rather fragile, no matter what...
"The application ran out of virtual memory. Please close some windows to free memory"
That's not a Photoshop error message.
It sounds like your system disk could be low on space, or your Windows paging file is set to too low a limit.
Chris, I still do not understand WHY this is happening. Even when I try 4 files that when uncompressed add up to only 231MB, it fails. This makes no sense. PS is reporting it can use 1.5GB of memory. I don't see why this would even be bothering with VM. It should be an in-memory operation. I also have a huge amount of paging space (about 12GB in total). This seems like overkill for the task at hand rather than being insufficient!
Is the photomerge not actually using real memory from that which has been covered out for PS from within PS. I started at 70% and increase it to 90%. I'm think I should try going the other way and reduce it to 40% or 50%, leaving more for Photomerge to use if this is the case.
Something about your system configuration is the cause -- but we don't know your complete system configuration.
EVERY bit of RAM allocated is also backed up by the OS VM.
So your paging file has to be at least as large as your RAM, preferably larger.
(The same goes for Photoshop's scratch -- it always has to allocate space on the disk for everything that is in memory)
It sounds like either your OS volume is full, or your paging limits are set too low for how you are using your system (other apps might also be using Gigs of space).
Checking the VM allocated in Task Manager would tell whether it is running out. If the 12GB of VM you have allocated is spread across multiple drives, then one of them running out might cause the error, although I'd expect you'd also get a low-disk warning message around the same time as the other warning.